In-vehicle systems

Abstract

To provide an on-vehicle system which can prevent vehicle control by feeling from not being performed.SOLUTION: An on-vehicle system comprises: a control device; an in-cabin sensor which detects whether or not a priority object person exists among occupants existing in a cabin; and a storage unit which stores a learned model for feeling estimation. The control device is configured to: determine whether or not the priority object person exists among the occupants existing in the cabin on the basis of a detection result of the in-cabin sensor; acquire information concerning the priority object person if it is determined that the priority object person exists; estimate feeling of the priority object person from the acquired information by using the learned model; and execute vehicle control according to an estimated result of the feeling of the priority object person.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an in-vehicle system. [Background technology]

[0002] Patent Document 1 discloses a technique for generating driving advice based on the difference in emotions between the driver and passengers. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-098779 Summary of the Invention [Problem to be solved by the invention]

[0004] When there are multiple occupants in the vehicle, the emotions of at least some of the occupants may differ from those of the other occupants, which may make it impossible to control the vehicle based on their emotions.

[0005] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an in-vehicle system that can prevent a loss of vehicle control due to emotions. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, the in-vehicle system of the present invention is an in-vehicle system comprising a control device, an in-vehicle sensor that detects whether a priority target person is present among the occupants in the vehicle, and a storage device that stores a trained model for emotion estimation, wherein the control device is configured to determine whether a priority target person is present among the occupants in the vehicle based on the detection result of the in-vehicle sensor, acquire information about the priority target person if it is determined that the priority target person is present, estimate the emotion of the priority target person from the acquired information using the trained model, and perform vehicle control according to the result of estimating the emotion of the priority target person.

[0007] As a result, in the in-vehicle system according to the present invention, when there are multiple occupants in the vehicle, the emotion estimation results used for vehicle control can be narrowed down to the estimation results of the priority target person, thereby preventing the in-vehicle system according to the present invention from becoming unable to control the vehicle based on emotions.

[0008] In the above, the priority target persons may be at least one of elderly people, children, people with disabilities, and people requiring care.

[0009] This allows for more comfortable vehicle control for those who are more likely to become fatigued by vehicle behavior than those who are generally physically fit to be designated as priority targets.

[0010] In addition, in the above, the vehicle may further include an imaging device positioned in a position capable of photographing multiple occupants, the trained model being generated by machine learning to derive an estimated result of a person's emotions from image data showing the person's facial expression, the information regarding the priority target person being composed of image data showing the facial expression of the priority target person obtained by the imaging device, and estimating the emotions of the priority target person using the trained model may be configured by providing the image data showing the facial expression of the priority target person obtained by the imaging device to the trained model, and performing computational processing of the trained model to obtain an estimated result of the emotions of the priority target person from the trained model.

[0011] This makes it possible to estimate emotions from the facial expressions of the priority subject in the image data captured by the imaging device.

[0012] In addition, in the above, the in-vehicle sensor may be configured to be composed of the imaging device, and determining whether or not the priority target person is present among the occupants present in the vehicle may be configured to determine whether or not the priority target person is present among the occupants present in the vehicle based on image data obtained by the imaging device.

[0013] This makes it possible to determine whether or not a priority person exists based on image data captured by the imaging device.

[0014] In addition, in the above, executing the vehicle control may include executing vehicle control that limits the range of acceleration when the result of estimating the emotion of the priority target person indicates that the priority target person is expressing unpleasant emotions.

[0015] This can prevent the priority passenger from feeling uncomfortable due to sudden acceleration or deceleration. [Effects of the Invention]

[0016] The in-vehicle system according to the present invention has an effect of preventing the vehicle from becoming uncontrollable due to emotions. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram showing an outline of a vehicle equipped with an in-vehicle system according to a first embodiment. [Figure 2] 4 is a flowchart showing an example of control performed by a control device. [Figure 3] FIG. 10 is a diagram showing an outline of a vehicle equipped with an in-vehicle system according to a second embodiment. [Figure 4] FIG. 10 is a diagram showing an outline of a vehicle equipped with an in-vehicle system according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] (Embodiment 1) A first embodiment of an in-vehicle system according to the present invention will be described below, although the present invention is not limited to this embodiment.

[0019] FIG. 1 is a diagram showing an outline of a vehicle 1 equipped with an in-vehicle system 2 according to the first embodiment.

[0020] 1, a vehicle 1 according to the embodiment includes an in-vehicle system 2, a steering wheel 4, front seats 31 and 32, and a rear seat 33. An arrow A in FIG. 1 indicates the traveling direction of the vehicle 1.

[0021] Occupants 10A, 10B, and 10C are seated in the front seats 31, 32 and the rear seat 33, respectively. Occupant 10A, seated in the front seat 31 opposite the steering wheel 4, is the driver of the vehicle 1. In the following description, when there is no need to distinguish between occupants 10A, 10B, and 10C, they will simply be referred to as occupant 10.

[0022] The in-vehicle system 2 is configured with a control device 21, a storage device 22, an in-vehicle camera 23, an operation panel 24, and the like.

[0023] The control device 21 is configured, for example, by an integrated circuit including a CPU (Central Processing Unit). The control device 21 is communicably connected to the storage device 22, the in-vehicle camera 23, and the operation panel 24. The control device 21 executes programs stored in the storage device 22. The control device 21 also acquires image data from the in-vehicle camera 23, for example.

[0024] The storage device 22 includes, for example, at least one of a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), and an HDD (Hard Disk Drive). Furthermore, the storage device 30 does not need to be a single physical element, but may include multiple elements that are physically separated from one another. The storage device 22 stores programs executed by the control device 21, etc. The storage device 22 also stores various data used when executing programs, such as trained machine learning models described below, such as a trained model for determining whether or not a vehicle is a weak driver, a trained model used for emotion estimation, and a trained model for vehicle control.

[0025] As shown in Fig. 1, the in-vehicle camera 23 is an imaging device located in a position inside the vehicle where it can capture images of multiple occupants 10A, 10B, and 10C. The in-vehicle camera 23 functions as an in-vehicle sensor for detecting priority vehicle-vulnerable occupants such as elderly people, children, people with disabilities, and people requiring care from the multiple occupants 10A, 10B, and 10C inside the vehicle, and outputs image data as sensor data. The image data captured by the in-vehicle camera 23 is transmitted to the control device 21 and temporarily stored in the storage device 22.

[0026] Here, "elderly" refers to a group of members in society who are older than other members, and the age standard may be defined as appropriate. In one example, elderly may be defined as those aged 65 or older. In another example, elderly may not be defined absolutely by age, but may be defined taking into account other factors such as physical ability (e.g., elderly may be defined as those whose physical ability declines with age). "Children" refers to a group of members in society who are younger than other members, and the age standard may be defined as appropriate. In one example, children may be defined as those under 18 years of age, under 15 years of age, under 12 years of age, or under 6 years of age. In another example, children may not be defined absolutely by age, but may be defined taking into account other factors such as physical ability (e.g., children may be defined as those who use child car seats). "Disabled" may be defined as appropriate to include at least one of physically disabled, intellectually disabled, and mentally disabled. In one example, a "person with a disability" may be defined as a person who is continually restricted in carrying out daily or social activities due to a lack of physical ability, etc. A "person requiring care" may be defined as a person who requires care. The scope of care need not be particularly limited and may be determined appropriately depending on the embodiment.

[0027] The operation panel 24 is an input / output device such as a touch panel display provided near the driver's seat, and receives operation instructions from the occupant 10 such as the driver, and provides information to the occupant 10.

[0028] The control device 21 can detect the attributes and facial expressions of the occupant 10 based on image data captured by the in-vehicle camera 23. That is, the control device 21 can determine the attributes of the occupant 10, such as a driver's disability, and the emotions of the occupant 10 based on the image data using AI (Artificial Intelligence) with a trained model learned through machine learning. Furthermore, the control device 21 can determine the content of vehicle control based on the emotions of the occupant 10 with AI using a trained model learned through machine learning.

[0029] The trained model for determining whether or not a person is a vulnerable driver is a trained machine learning model that has been machine-learned, for example, through supervised learning according to a neural network model, to output a determination result of whether or not the person is a vulnerable driver from input data. The trained model for determination is generated by repeatedly executing a learning process using a training dataset, which is a combination of input and result data. The training dataset includes, for example, multiple pieces of training data in which input data, such as the appearance of the occupant 10 and whether or not the occupant uses a wheelchair, is labeled as an output indicating whether or not the occupant is a vulnerable driver. The labeling of the input data as whether or not the occupant is a vulnerable driver is performed, for example, by a person skilled in the art. In this way, when the trained model for determination, trained using the training dataset, receives input data, it executes arithmetic processing of the trained model to output whether or not the occupant is a vulnerable driver.

[0030] The trained model for emotion estimation is a trained machine learning model that has been trained, for example, by supervised learning according to a neural network model, to output emotion estimation results from input data. The training dataset in the trained model for emotion estimation includes, for example, a plurality of training data in which input data, such as image data showing a person's facial expression, such as the facial expression of the occupant 10, is labeled with the emotion of the occupant 10 to be output. The labeling of the emotion of the occupant 10 to the input data is performed, for example, by a person skilled in the art. In this way, the trained model for emotion estimation trained using the training dataset receives input data and outputs the emotion of the occupant 10 by executing arithmetic processing of the trained model.

[0031] The data used to determine whether the occupant 10 is a weak driver or not and the data used to estimate the emotion of the occupant 10 may be the same or different.

[0032] The trained vehicle control model is a trained machine learning model that has been trained, for example, through supervised learning according to a neural network model, to output vehicle control results from input data. The training dataset in the trained vehicle control model includes multiple training data labeled with output vehicle control results for input data, such as the emotions of the occupant 10. Labeling of the vehicle control results for the input data is performed, for example, by a person skilled in the art. Upon receiving input data, the trained vehicle control model trained using the training dataset executes computational processing of the trained model to output vehicle control results. Examples of vehicle control results include limiting the range of acceleration or ensuring that the steering angle and lateral G are below thresholds. Furthermore, examples of vehicle control results include softening the suspension of the vehicle 1 to improve ride comfort when an elderly person is detected as a vulnerable driver. The execution of vehicle control by the control device 21 includes executing vehicle control that limits the range of acceleration, etc., when the result of estimating the emotions of the priority target person indicates that the priority target person is expressing unpleasant emotions.

[0033] In addition, when determining the content of vehicle control, the control device 21 may determine the content of vehicle control from the emotions of the occupant 10 based on rules that associate the emotions of the occupant 10 with the content of vehicle control, rather than using a learned model for vehicle control.

[0034] The control device 21 determines whether or not a vulnerable driver, such as an elderly person, a child, or a disabled person, is in the vehicle, and identifies the vehicle's seating position, based on the detection results of in-vehicle sensors such as the in-vehicle camera 23. The control device 21 determines whether or not a vulnerable driver, who is a priority target, is present among the occupants 10 present in the vehicle by determining whether or not a vulnerable driver is present among the occupants 10 present in the vehicle, based on image data obtained by the in-vehicle camera 3.

[0035] Here, compared to physically fit individuals, weak drivers tend to become fatigued more easily due to the behavior of the vehicle 1. Compared to other occupants 10, weak drivers have a harder time understanding the behavior of the vehicle 1, such as acceleration / deceleration, turning, and bumps, and are more likely to feel uncomfortable due to the behavior of the vehicle 1. Therefore, in the in-vehicle system 2 according to the first embodiment, the control device 21 determines whether a weak driver is present among the multiple occupants 10 of the vehicle 1 based on image data captured by the in-vehicle camera 23. If a weak driver is present in the vehicle 1, the control device 21 performs a process of identifying and marking the riding position (seating position) of the weak driver. The control device 21 then acquires information about the marked weak driver, prioritizes emotion estimation of the weak driver, and executes vehicle control to limit acceleration, steering angle, and the like based on the emotion estimation results. The information about the weak driver consists, for example, of image data showing the facial expressions of the weak driver, obtained by the in-vehicle camera 3.

[0036] The priority of emotion estimation for those with poor driving skills may be determined by seat position, with infants, small children, the elderly, wheelchair users, pregnant women, and people with disabilities all being treated in the same order regardless of their attributes. For example, the priority of emotion estimation may be higher for those with poor driving skills who are seated in the rear seat 33, which is more susceptible to car sickness, than for those in the front seats 31 and 32. Furthermore, if the driver of vehicle 1 is a poor driver, such as an elderly person, the priority of emotion estimation for the driver may be lower.

[0037] Furthermore, the control device 21 may use the in-vehicle camera 23 to determine whether or not the occupant 10 is a weak driver based on the behavior of the occupant 10 when getting into the vehicle 1. For example, when the control device 21 detects, based on image data captured by the in-vehicle camera 23, that the occupant 10 is getting into the vehicle from a wheelchair or that the occupant 10 is taking a little time to get into the vehicle, the control device 21 determines that the occupant 10 is a weak driver.

[0038] In addition, in the in-vehicle system 2 according to the first embodiment, the seat positions inside the vehicle may be displayed on the display of the operation panel 24, and the occupant 10, such as the driver, may specify the seat position where the car-vulnerable person, who is the subject for whom emotion estimation is prioritized, is sitting.

[0039] Furthermore, in the in-vehicle system 2 according to the first embodiment, for example, it is possible to determine the weak car users by targeting only elderly people or only occupants 10 sitting in the rear seats 33, and to perform emotion estimation by narrowing down the priority, thereby reducing the processing load on the control device 21.

[0040] Furthermore, in the in-vehicle system 2 according to the first embodiment, it may be determined whether the driver is feeling "comfortable" or "uncomfortable" based on the behavior of the vehicle 1, and the determination may be reflected in vehicle control.

[0041] Furthermore, in the in-vehicle system 2 according to the first embodiment, if there is no priority target vehicle-vulnerable person, the result of emotion estimation to be used for vehicle control may be selected by any method, or vehicle control based on the result of emotion estimation may not be performed.

[0042] FIG. 2 is a flowchart showing an example of the control performed by the control device 21.

[0043] First, in step S1, the control device 21 acquires image data captured by the in-vehicle camera 23. Next, in step S2, the control device 21 determines whether or not a vulnerable vehicle user, who is a priority target, is present among the multiple occupants 10 in the vehicle. If the control device 21 determines that no vulnerable vehicle user is present among the occupants 10, the control device 21 determines No in step S2 and terminates the series of control operations. On the other hand, if the control device 21 determines that a vulnerable vehicle user is present among the occupants 10, the control device 21 determines Yes in step S2 and proceeds to step S3. In step S3, the control device 21 detects the facial expression of the vulnerable vehicle user from the image data captured by the in-vehicle camera 23 and acquires information about the vulnerable vehicle user. Next, in step S4, the control device 21 estimates the emotion of the vulnerable vehicle user based on the detected facial expression using a trained model for emotion estimation. Next, in step S5, the control device 21 determines the content of vehicle control based on the estimated emotion using a trained model for vehicle control. Next, in step S6, the control device 21 executes vehicle control based on the determined content of vehicle control. Thereafter, the control device 21 ends the series of controls.

[0044] The in-vehicle system 2 according to the first embodiment can prevent the poor driver from feeling uncomfortable by controlling the vehicle while giving priority to the emotions of the poor driver, who is a priority target. Furthermore, in the in-vehicle system 2 according to the first embodiment, when there are multiple occupants 10 in the vehicle, the estimation result of the emotions used for vehicle control can be narrowed down to the estimation result of the poor driver, who is a priority target. Therefore, the in-vehicle system 2 according to the first embodiment can prevent the vehicle from being unable to be controlled based on emotions.

[0045] (Embodiment 2) Hereinafter, a second embodiment of the in-vehicle system according to the present invention will be described. Note that the description of the second embodiment common to the first embodiment will be omitted as appropriate.

[0046] FIG. 3 is a diagram showing an outline of a vehicle 1 equipped with an in-vehicle system 2 according to the second embodiment.

[0047] In the vehicle 1 according to the second embodiment, a child occupant 10D is sitting in a child seat 34 installed in the rear seat 33. In this case, the control device 21 in the in-vehicle system 2 according to the second embodiment detects the child seat 34 and the child occupant 10D sitting in the child seat 34 based on image data from the in-vehicle camera 23, thereby determining that a vulnerable vehicle occupant is present in the vehicle. Note that the learning dataset in the trained model for determination includes, for example, a plurality of learning data items that are output and labeled as to whether or not the occupant 10 is a vulnerable vehicle occupant, in response to input data indicating whether or not the occupant 10 is sitting in a child seat.

[0048] A child occupant 10D, who is a weak driver and is seated in a child car seat 34, is prone to feeling uncomfortable because he or she cannot predict the behavior of the vehicle 1, such as acceleration / deceleration, turning right or left, and going over bumps. Therefore, the control device 21 preferentially estimates the emotion of the child occupant 10D, who is a weak driver and is seated in a child car seat 34, based on image data captured by the in-vehicle camera 23, using a trained model for emotion estimation from the facial expression of the occupant 10D. Then, the control device 21 executes vehicle control that limits acceleration and the like using a trained model for vehicle control based on the result of emotion estimation.

[0049] Furthermore, in the in-vehicle system 2 according to the second embodiment, a seat belt sensor that detects whether a seat belt is fastened may be used as an in-vehicle sensor for detecting a vulnerable vehicle occupant. The control device 21 determines whether the occupant 10 is a priority target for emotion estimation based on whether a seat belt is fastened at the seating position of the occupant 10. For example, the control device 21 detects whether a seat belt provided at the seating position of the rear seat 33 where a child seat 34 is installed is fastened. When the control device 21 detects that the seat belt provided at that seating position is not fastened, the control device 21 determines that the occupant 10D at that seating position is a child sitting in the child seat 34, and detects a vulnerable vehicle occupant. Furthermore, in the in-vehicle system 2 according to the second embodiment, the control device 21 may detect whether a child is sitting in the child seat 34 based on the detection result of a weight sensor provided at the seating position of the rear seat 33 where the child seat 34 is installed, thereby detecting a vulnerable vehicle occupant.

[0050] (Embodiment 3) Hereinafter, an in-vehicle system according to a third embodiment of the present invention will be described. Note that the description of the content common to the first embodiment of the third embodiment will be omitted as appropriate.

[0051] FIG. 4 is a diagram showing an outline of a vehicle 1 equipped with an in-vehicle system 2 according to the third embodiment.

[0052] In the vehicle 1 according to the third embodiment, a wheelchair space is provided instead of a rear seat, and an occupant 10E is seated in a wheelchair 35. The wheelchair 35 is secured inside the vehicle by a locking device 25. In this case, in the in-vehicle system 2 according to the third embodiment, the control device 21 detects the occupant 10E seated in the wheelchair 35 based on image data from the in-vehicle camera 23, and thereby determines that a vulnerable vehicle user is present inside the vehicle. Note that the training data set in the trained model for determination includes, for example, a plurality of training data in which input data indicating whether the occupant 10 is sitting in a wheelchair is labeled as an output indicating whether the occupant is a vulnerable vehicle user.

[0053] Occupant 10E sitting in wheelchair 35 is prone to feeling uncomfortable because he or she cannot predict the behavior of vehicle 1, such as acceleration / deceleration, turning right or left, and going over steps. Therefore, control device 21 preferentially estimates the emotion of occupant 10E sitting in wheelchair 35, who is a weak driver, using a trained model for emotion estimation based on the facial expression of occupant 10E, based on image data captured by in-vehicle camera 23. Then, control device 21 executes vehicle control that limits acceleration and the like, using a trained model for vehicle control, based on the result of emotion estimation.

[0054] Furthermore, in the in-vehicle system 2 according to the third embodiment, for example, a lock sensor that detects whether or not the wheelchair 35 is locked by the locking device 25 may be used as an in-vehicle sensor for detecting the vehicle-vulnerable person. This lock sensor is provided in the locking device 25 and is communicably connected to the control device 21. Then, for example, when the lock sensor detects that the wheelchair 35 is locked by the locking device 25, the control device 21 determines that the occupant 10E sitting in the wheelchair 35 is in the vehicle 1, and detects the vehicle-vulnerable person. [Explanation of symbols]

[0055] 1 vehicle 2. In-vehicle systems 4 Handle 10, 10A, 10B, 10C, 10D, 10E crew 21 Control device 22 Storage device 23 In-car camera 24 Operation Panel 25 Locking device 31,32 Front seats 33 Back seat 34 Child Seat 35 Wheelchair

Claims

1. a control device; an in-vehicle sensor that detects whether or not a priority person is present among the occupants present in the vehicle; a storage device that stores a trained model for emotion estimation; An in-vehicle system comprising: The control device determining whether or not a priority person is present among the occupants present in the vehicle based on the detection result of the in-vehicle sensor; If it is determined that the priority target person exists, only information about the priority target person is acquired; Using the trained model, estimate the emotion of the priority target person from the acquired information, and Execute vehicle control in accordance with the result of estimating the emotion of the priority target person; not executing vehicle control when it is determined that the priority target person does not exist; It is configured as follows: In-vehicle systems.

2. The priority target persons are at least one of elderly people, children, people with disabilities, and people requiring care. The in-vehicle system according to claim 1 .

3. The vehicle further includes an imaging device disposed in a position capable of capturing an image of the plurality of occupants in the vehicle, the trained model is generated by machine learning to derive a result of estimating a person's emotion from image data showing the person's facial expression, the information about the priority target person is configured by image data showing a facial expression of the priority target person obtained by the imaging device, Estimating the emotion of the priority target person using the trained model includes: Image data showing the facial expression of the priority target person obtained by the imaging device is provided to the trained model, and Executing a calculation process of the trained model to obtain an estimation result of the emotion of the priority target person from the trained model; It consists of The in-vehicle system according to claim 1 .

4. the in-vehicle sensor is configured by the imaging device, determining whether or not the priority person is present among the occupants present in the vehicle is configured to determine whether or not the priority person is present among the occupants present in the vehicle based on image data obtained by the imaging device; The in-vehicle system according to claim 3 .

5. executing the vehicle control includes executing vehicle control to limit a range of acceleration when a result of estimating the emotion of the priority target person indicates that the priority target person is expressing an unpleasant emotion. The in-vehicle system according to any one of claims 1 to 4.

Images