Device for moving body and control method for moving body

Abstract

The HCU (20) that can be used in a vehicle includes: an occupant information determination unit (206) that determines occupant information, i.e., occupant information, detected by sensors used in the vehicle in a manner that distinguishes each occupant of the vehicle; and a request estimation unit (208) that estimates a request for an occupant corresponding to a combination of occupant information of multiple occupants of the vehicle based on the occupant information of multiple occupants determined by the occupant information determination unit (206).

Description

[0001] Cross-references of related applications

[0002] This application is based on Japanese Patent Application No. 2021-106090, filed in Japan on June 25, 2021, and incorporates the contents of the base application by reference in its entirety. Technical Field

[0003] This disclosure relates to a device for moving bodies and a control method for moving bodies. Background Technology

[0004] Patent Document 1 discloses the following technology: obtaining personal information corresponding to the occupants of a vehicle from a personal information database, and then providing information based on the obtained personal information to the occupants of the vehicle.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-155570

[0006] For example, occupants' requests vary depending on their relationships and statuses. Therefore, it is difficult to accurately predict occupants' requests based solely on their personal information. Furthermore, because it is difficult to accurately predict occupants' requests, it is challenging to provide a comfortable in-vehicle experience that matches those requests. Summary of the Invention

[0007] One of the objectives of this disclosure is to provide a mobile body device and a mobile body control method that can provide a more comfortable indoor experience that matches the occupants' requests by more accurately inferring the requests of multiple occupants in the presence of multiple occupants.

[0008] The aforementioned objective is achieved through a combination of features described in the independent claims. Furthermore, the subordinate claims specify more advantageous examples. The reference numerals within parentheses in the claims indicate the correspondence between specific units described in the embodiments described later as part of a technical solution, and do not limit the scope of this disclosure.

[0009] To achieve the above objectives, the mobile body device disclosed herein can be used in a mobile body, wherein the mobile body device includes: an occupant information determination unit that determines occupant information, i.e., occupant information, detected by sensors used in the mobile body in a manner that distinguishes each occupant of the mobile body; and a request estimation unit that, based on the occupant information of multiple occupants of the mobile body determined by the occupant information determination unit, estimates a request for an occupant corresponding to a combination of the occupant information of the multiple occupants.

[0010] To achieve the above objectives, the mobile body control method disclosed herein can be used in a mobile body, wherein the mobile body control method includes the following steps executed by at least one processor: an occupant information determination step, determining occupant information, i.e., occupant information, detected by sensors used in the mobile body in a manner that distinguishes individual occupants of the mobile body; and a request estimation step, estimating a request for an occupant corresponding to a combination of occupant information of multiple occupants of the mobile body based on the occupant information of multiple occupants of the mobile body determined in the occupant information determination step.

[0011] Therefore, based on occupant information of multiple occupants of a mobile body, determined in a way that distinguishes each occupant from the others, the requests of the occupants corresponding to combinations of this information can be inferred. This allows for more accurate inference of occupant requests that vary depending on the relationships and states of the multiple occupants. Furthermore, since the occupant information is detected by sensors used within the mobile body, occupant requests that match the actual situation can be inferred with greater accuracy. As a result, by inferring occupant requests in the presence of multiple occupants with greater accuracy, a more comfortable indoor experience that better matches the occupant's request can be provided. Attached Figure Description

[0012] Figure 1 This is a diagram showing an example of the general structure of system 1 for vehicles.

[0013] Figure 2 This is a diagram illustrating an example of the general structure of HCU20.

[0014] Figure 3 This is a flowchart illustrating an example of a process that provides associated processing in HCU20. Detailed Implementation

[0015] The various embodiments disclosed will be described with reference to the accompanying drawings. Furthermore, for ease of explanation, parts having the same function as those shown in the figures used in the preceding description are labeled with the same reference numerals, and their descriptions are sometimes omitted. For parts labeled with the same reference numerals, reference can be made to the descriptions in other embodiments.

[0016] (Implementation Method 1)

[0017] <Brief Structure of System 1 for Vehicles>

[0018] Hereinafter, this embodiment will be described using the accompanying drawings. Figure 1The vehicle system 1 shown is described as an example of a structure used in an automobile (hereinafter referred to as a vehicle). Vehicle system 1 includes an HMI (Human Machine Interface) system 2, an air conditioning system 3, a near-field communication module (hereinafter, NFCM) 4, a wide-area communication module (hereinafter, WACM) 5, and a seat ECU 6. The HMI system 2, air conditioning system 3, NFCM 4, WACM 5, and seat ECU 6 are connected, for example, to an in-vehicle LAN. Hereinafter, a vehicle equipped with vehicle system 1 will be referred to as this vehicle.

[0019] The air conditioning system 3 is a vehicle-use cooling and heating system. The air conditioning system 3 obtains air conditioning request information, including settings related to air conditioning configured by the vehicle's user, from the HCU 20 (described later). Furthermore, based on the obtained air conditioning request information, it adjusts the temperature, airflow, and fragrance within the vehicle's interior. The air conditioning system 3 includes an air conditioning control ECU 30, an air conditioning unit 31, and a fragrance unit 32.

[0020] The air conditioning unit 31 generates both warm and cold air (hereinafter, air conditioning air). The air conditioning air is supplied to the vehicle interior through an outlet located in the vehicle, such as the dashboard. The fragrance unit 32 has beads or the like (hereinafter, impregnated material) impregnated with essential oils containing aromatic components. Furthermore, the fragrance is supplied to the vehicle interior by passing the airflow generated by the air conditioning unit 31 around the impregnated material. Alternatively, the fragrance unit 32 can also atomize the aromatic oil. In this case, a structure is adopted where the aromatic components atomized by the fragrance unit 32 are mixed with the airflow generated by the air conditioning unit 31 and supplied to the vehicle interior. The air conditioning unit 31 provides wind-based stimulation to the occupants of the vehicle. Additionally, the air conditioning unit 31 provides hot and cold stimulation to the occupants of the vehicle based on the temperature difference of the air conditioning air. That is, the air conditioning unit 31 provides tactile stimulation. The fragrance unit 32 provides fragrance-based stimulation to the occupants of the vehicle. That is, the fragrance unit 32 provides olfactory stimulation. Both the air conditioning unit 31 and the aromatherapy unit 32 are cue devices that provide stimulation. The air conditioning control ECU 30 is an electronic control device that controls the operation of the air conditioning unit 31 and the aromatherapy unit 32. The air conditioning control ECU 30 is connected to the air conditioning unit 31 and the aromatherapy unit 32.

[0021] NFCM4 is a communication module used for short-range wireless communication. Once a communication connection is established with a mobile terminal belonging to a passenger in the vehicle, NFCM4 conducts short-range wireless communication with that mobile terminal. Short-range wireless communication refers to wireless communication with a maximum convergence range of, for example, around tens of meters. Examples of short-range wireless communication include Bluetooth (registered trademark) low-energy wireless communication. Examples of mobile terminals include multi-functional mobile phones and wearable devices. WACM5 transmits and receives information wirelessly with an external hub of the vehicle, i.e., performs wide-area communication.

[0022] The seat ECU6 is an electronic control device that performs various processes related to controlling the seating environment, such as adjusting the seat position of the vehicle's seats. Here, we will explain this as an electric seat capable of electrically changing its sliding and reclining positions. If the vehicle's seats are not electric seats, the seat ECU6 can be omitted. Examples of seats include the driver's seat, front passenger seat, and rear seats. An electric seat may also be only a part of the driver's seat, front passenger seat, or rear seats. The sliding position refers to the seat's position in the fore-and-aft direction of the vehicle. The reclining position refers to the angle of the seat back. The seat back can also be referred to as the seat back.

[0023] HMI System 2 acquires information from the occupant or provides stimuli to the occupant. These stimuli include the provision of information. Details about HMI System 2 are described below.

[0024] <Overview of HMI System 2>

[0025] The HMI system 2 includes an HCU (Human Machine Interface Control Unit) 20, an indoor camera 21, a microphone 22, a lighting device 23, a display device 24, and a sound output device 25.

[0026] Interior camera 21 captures a defined area inside the vehicle interior. Interior camera 21 captures the area including the driver's seat, front passenger seat, and rear seats. Multiple cameras can also be used as interior camera 21 to share the capture area. Interior camera 21 consists, for example, of a near-infrared light source, a near-infrared camera, and a control unit that controls them. Interior camera 21 captures images of occupants illuminated by the near-infrared light source using the near-infrared camera. The control unit performs image analysis on the images captured by the near-infrared camera. Based on the occupant features extracted from the image analysis, the control unit detects the occupant's alertness, facial orientation, gaze direction, posture, etc. Awakeness can be detected, for example, based on the degree of eyelid opening and closing.

[0027] Microphone 22 collects the sounds emitted by the occupants of the vehicle, converts them into electroacoustic signals, and outputs them to HCU 20. Preferably, microphone 22 is installed for each seat to allow for the differentiation of the sounds emitted by occupants in different seats. However, if the occupant's structure is determined through voice recognition (described later), microphone 22 may not be installed for each seat. As the microphone 22 installed for each seat, a zoom microphone with reduced directional focus is sufficient.

[0028] The lighting device 23 is positioned at a location that the occupant can visually recognize, providing the occupant with light-based stimulation. That is, it provides visual stimulation. The lighting device 23 is a cueing device that provides this stimulus. As the lighting device 23, a light-emitting device such as an LED can be used. Preferably, the lighting device 23 is capable of switching the color of its emitted light. The illumination of the lighting device 23 is controlled by the HCU 20.

[0029] Display device 24 displays information. Display device 24 is positioned in a location where the occupant can visually confirm it, providing the occupant with display-based stimulation. That is, providing visual stimulation. Display device 24 is a cueing device for cueing stimuli. Preferably, display device 24 displays at least an image. In addition to displaying images, display device 24 may also display text, etc. The display of display device 24 is controlled by HCU 20. For example, instrument panel MID (Multi Information Display), CID (Center Information Display), rear-seat display, transparent display, and transmissive skin display can be used as display device 24.

[0030] An instrument cluster display (MID) is a display device located in front of the driver's seat inside the vehicle. For example, an instrument cluster MID can be located on the instrument panel. A center display (CID) is a display device located in the center of the vehicle's dashboard. A rear seat display is a display device facing the rear occupants of the vehicle. A rear seat display can be located on the back of the driver's or front passenger seat, or in the headliner, with the display facing the rear of the vehicle. A transparent display is a transmissive display device. Examples of transparent displays include OLEDs (Organic Electroluminescence). A transparent display can be located on a window of the vehicle. A transmissive skin display is a display device that displays information through a transmissive skin. A transmissive skin display can be located on the door panels, seatbacks, floor, or roof of the vehicle.

[0031] The sound output device 25 provides sound-based stimulation to the occupant. That is, it provides auditory stimulation. The sound output device 25 is a cueing device for cues. Examples of sounds output from the sound output device 25 include music and ambient sounds. Music may also include background music (BGM). Ambient sounds can reproduce the sounds of a specific environment. The sound output device 25 can be, for example, an audio speaker that outputs sound.

[0032] The HCU20 is primarily composed of a microcomputer equipped with a processor, memory, I / O, and a bus connecting them. The HCU20 executes various processes related to providing the interior environment of the vehicle (hereinafter, provision-related processing) by executing control programs stored in the memory. This HCU20 is equivalent to a mobile device. The memory referred to here is a non-transitory tangible storage medium that non-transitory stores programs and data that can be read by a computer. Furthermore, the non-transitory tangible storage medium is implemented using semiconductor memory or a hard disk, etc. The general structure of the HCU20 is described below.

[0033] <Brief Structure of HCU20>

[0034] Next, use Figure 2 The general structure of HCU20 is explained below. For example... Figure 2 As shown, the HCU20 includes occupant authentication unit 201, authentication database (hereinafter, DB) 202, provision processing unit 203, voice recognition unit 204, personal DB 205, occupant information determination unit 206, auxiliary information acquisition unit 207, request estimation unit 208, and indoor environment determination unit 209 as functional modules. Furthermore, the processing of these functional modules by a computer is equivalent to executing a mobile body control method. In addition, some or all of the functions performed by the HCU20 can also be implemented in hardware by one or more ICs, etc. Furthermore, some or all of the functional modules of the HCU20 can also be implemented through a combination of processor-based software execution and hardware components.

[0035] The occupant authentication unit 201 performs authentication of the vehicle's registered occupants. Authentication is performed by comparing the information with that of registered occupants pre-registered in the authentication DB 202. The authentication DB 202 can be a non-volatile memory. Alternatively, the authentication DB 202 can be an authentication DB located in a center capable of communication via WACM5. It is preferable to use multiple authentication methods. For example, authentication utilizing the vehicle's sensors and authentication based on cooperation with the occupants' mobile terminals are preferred.

[0036] As an example, for authentication utilizing the vehicle's sensors, facial authentication using facial features detected from images captured by the interior camera 21 is listed. Additionally, iris authentication using iris features detected from the captured images is listed. Furthermore, when fingerprint sensors are installed on the vehicle's door handles, fingerprint authentication using fingerprints detected by those sensors is also listed. For authentication based on cooperation with the vehicle's occupants' mobile terminals, code matching using individual occupant identification information held by the mobile terminals is listed. The occupant authentication unit obtains this identification information from the mobile terminals via NFCM4. Besides authentication, the occupant authentication unit 201 can also determine whether an occupant is the driver or a passenger based on their seating position. The seating position can be detected by a seating sensor or inferred from the detection of open or closed doors. Furthermore, each occupant can be identified based on the identification information held by the mobile terminals.

[0037] The occupant authentication unit 201 can, for example, use multiple authentication methods to improve authentication accuracy. Specifically, the successful authentication of one of the multiple authentication methods can be set as a condition for successful authentication. Furthermore, the occupant authentication unit 201 can, for example, use multiple authentication methods for faster authentication. Specifically, it can also adjust the seat position of each occupant and start the air conditioning system before the occupant boards the vehicle through authentication based on cooperation with the occupant's mobile terminal.

[0038] The processing unit 203 provides various interior environments by controlling various prompting devices in the vehicle. For example, when the occupant authentication is successful in the occupant authentication unit 201, the processing unit 203 can provide an interior environment suitable for a performance. Examples of an interior environment suitable for a performance include adjusting the seat positions of each occupant, displaying images related to the performance, providing lighting related to the performance, and outputting sound related to the performance. Adjusting the seat positions of each occupant can be achieved by instructing the seat ECU 6. Determining the seat positions of each occupant can be achieved by pre-storing and associating the identification information of the mobile terminal with the seat positions of each occupant. Lighting can be achieved by controlling the lighting device 23. Image display can be achieved by controlling the display device 24. Sound output can be achieved by controlling the sound output device 25.

[0039] The voice recognition unit 204 performs voice recognition on the sound collected by the microphone 22 to identify the content of the passenger's speech. Furthermore, if a microphone 22 is provided for each seat, the voice recognition unit 204 can differentiate and determine the content of each passenger's speech based on the different microphones 22 used for collection. Similarly, even if a microphone 22 is not provided for each seat, the voice recognition unit 204 can still differentiate and determine the content of each passenger's speech.

[0040] The personal DB205 pre-stores information about each crew member. Non-volatile memory can be used for the personal DB205. The information about each crew member includes information for identifying each crew member (hereinafter, crew member identification information). In addition, the personal DB205 also includes at least one of the following: information about each crew member's preferences and information about their past operational history (hereinafter, auxiliary information). Furthermore, the personal DB205 can also be a personal DB located in a center capable of communication via WACM5.

[0041] Information used to identify passengers can be obtained by associating each passenger's attributes, biometric information, and mobile device identification information. Passenger attributes here refer to kinship relationships based on defined individuals. The defined individuals will be referred to as "the person" in the following explanation. Passenger attributes include the passenger themselves, their spouse, grandparents, sons (and younger than a certain age), daughters (and younger than a certain age), infants (and younger than a certain age), their friends, their spouse's friends, their son's friends, and their daughter's friends. Furthermore, passenger attributes can be attributes other than those described here, or more granular attributes. For example, if there are multiple grandparents, sons, daughters, infants, and friends, they can be distinguished. For example, they can be designated as friend A, friend B, friend C, etc. Additionally, biometric information for passengers includes features extracted from facial images and voiceprints. As supplementary information, for example, for each crew member, at least one of the crew member's hobbies information (hereinafter, hobbies information) and past action history information (hereinafter, action history information) can be associated.

[0042] The occupant information determination unit 206 determines the occupant information (hereinafter, occupant information) detected by the sensors used in the vehicle in a manner that distinguishes each occupant of the vehicle. The processing in the occupant information determination unit 206 is equivalent to the occupant information determination process. Preferably, when determining each occupant of the vehicle, the attributes of each occupant are also determined. As a result, the estimation accuracy in the request estimation unit 208 described later is further improved. As the sensors mentioned here, examples include the interior camera 21 and the microphone 22. As occupant information, there is the speech content recognized by the voice recognition unit 204. As occupant information, there is information from the image of the occupant detected by the interior camera 21 (hereinafter, image source information). As image source information, there is at least one occupant state among the occupant's facial image detected by the interior camera 21, the feature quantity extracted from the facial image, and the occupant's action and posture detected by the interior camera 21. As occupant states, examples include the occupant's alertness, the occupant's facial orientation, the occupant's gaze direction, and the occupant's posture. Regarding the content of the speech, the occupant information determination unit 206 determines which occupant is speaking based on their voiceprint and attribute determination information stored in the personal database 205. Regarding image source information, the occupant information determination unit 206 determines which occupant's image source information is based on features extracted from a facial image and attribute determination information stored in the personal database 205. Furthermore, the occupant's seating position is determined using information obtained by the occupant authentication unit 201. In addition, regarding the content of the speech, if a microphone 22 is provided for each seat, the occupant's seating position is determined based on the speech content identified from the sound collected by that microphone 22. Furthermore, the methods for distinguishing individual occupants to determine speech content and image source information are not limited to the methods described above.

[0043] In this embodiment, a structure is shown in which the occupant's alertness, facial orientation, gaze direction, and posture are detected by the indoor camera 21 to determine the occupant's state, but this is not a limitation. For example, the indoor camera 21 may infer the occupant's psychological state based on feature values ​​from the occupant's facial image to determine the occupant's state. Alternatively, the occupant information determination unit 206 may be configured to perform the functions of detecting the occupant's alertness, facial orientation, gaze direction, posture, and psychological state, instead of the indoor camera 21 detecting the occupant's alertness, facial orientation, gaze direction, posture, and psychological state.

[0044] The auxiliary information acquisition unit 207 acquires the aforementioned auxiliary information. The auxiliary information acquisition unit 207 can acquire auxiliary information from the personal DB205. If the personal DB205 is located at a central point on the exterior of the vehicle, the auxiliary information acquisition unit 207 can acquire auxiliary information from that central point via the WACM5.

[0045] The request estimation unit 208 estimates the requests of passengers corresponding to combinations of passenger information of multiple passengers in the vehicle, based on passenger information determined by the passenger information determination unit 206. This processing in the request estimation unit 208 is equivalent to a request estimation step. The request estimation unit 208 can estimate requests for all multiple passengers in the vehicle, or it can estimate requests for a subset of passengers. The request estimation unit 208 uses a machine learning learner to estimate the requests of passengers corresponding to combinations of passenger information of multiple passengers based on the passenger information of multiple passengers. In this case, the learner can be a machine learning learner obtained by taking the combination of passenger information of multiple passengers as input and the request of the passenger corresponding to that combination as output. Furthermore, the request estimation unit 208 can also estimate passenger requests based on the correspondence between the combination of passenger information of multiple passengers and the passenger requests estimated based on that combination. This correspondence can be obtained by listening to multiple subjects, etc. The following explanation will continue with the example of the request for the estimation unit 208 to estimate the crew member's request using the aforementioned learner.

[0046] The request estimation unit 208 infers the requests of passengers corresponding to combinations of the speech content of multiple passengers in the vehicle, as determined by the passenger information determination unit 206. In this case, a learner that has undergone machine learning is preferred, where the order of speech content is also used as input to infer the requests of passengers corresponding to the flow of speech content of multiple passengers, i.e., the conversation content. Even for speech content that has not been fully learned, the learner can infer the requests of passengers corresponding to combinations of speech content of multiple passengers based on the similarity of the elements of the speech content. Preferably, the request estimation unit 208 infers the background of the conversation content, as determined by the passenger information determination unit 206, and infers the requests of passengers matching that background. In this case, the inference can be performed in stages, such as inferring the background based on the conversation content and inferring the requests of passengers based on the inferred background. Therefore, even requests from crew members that are difficult to infer simply by understanding the content of multiple crew members' speeches can be inferred with greater accuracy. The "background" mentioned here can be translated as context or context. That is, the "background" can also be referred to as the literary context, the background, the situation, etc.

[0047] Furthermore, the preferred request estimation unit 208 estimates the requests of the occupants of the vehicle based not only on the occupant information of multiple occupants determined by the occupant information determination unit 206, but also on the auxiliary information acquired by the auxiliary information acquisition unit 207. In this case, a learner that performs machine learning can be used as the learner, in which the requests of occupants corresponding to the auxiliary information are estimated by also taking the auxiliary information as input. Thus, by also considering the occupants' preferences and past behavior history, the occupants' requests can be estimated with higher accuracy.

[0048] Here, we will explain an example of a passenger request corresponding to a combination of statements from multiple passengers. Let's assume the following is scenario A. Assume the multiple passengers who spoke are the wife in the front passenger seat, the son in the rear seat, and the daughter in the rear seat. Assume the speaking order is wife, son, then daughter. Assume the wife's statement is, "I didn't see any fireflies this year, what a pity." Assume the son's statement is, "Yes! They're nearby! Really. They were so beautiful last year." Assume the daughter's statement is, "Yes! I want to see them again." Assume the passenger information determination unit 206 distinguishes whose statements the wife, son, and daughter made.

[0049] In the request estimation unit 208, based on the order and content of the speech, it is inferred that the wife in the front passenger seat is talking to her son and daughter in the rear seats. Furthermore, it is possible to infer not only the content of the speech but also the wife's facial orientation as determined by the occupant information determination unit 206. Additionally, based on strings such as "didn't see fireflies" and "what a pity," it is inferred that the wife is feeling disappointed because she couldn't see fireflies. Furthermore, based on strings such as the son's "Really?" and the daughter's "Yes! I want to see them again," it is inferred that all three are feeling disappointed because they couldn't see fireflies. Moreover, based on this inference, it is inferred that a passenger who wants to see fireflies is making a request. In scenario A, a passenger who wants to see fireflies is inferred.

[0050] Furthermore, based on the son's message, "It's nearby!", it can be inferred that the current location is near a firefly habitat. Moreover, based on this inference, the wife's activity log can be used to extract photos taken in the vicinity of the current location last year, including photos of fireflies. Additionally, by using last year's activity log of taking photos of fireflies in the vicinity, the inference regarding the crew's request to see fireflies can be strengthened.

[0051] As another example, consider scenario B below. Let the multiple passengers who spoke be the wife in the driver's seat and the wife's friend in the passenger seat. Let the speaking order be the wife, the wife's friend, and then the wife again. Let the wife's first speech be: "The apple pie shop that recently opened in ZZ Shopping Center is delicious. I had it with my family a few days ago, and it was very popular." Let the wife's friend's speech be: "Hey! I'm curious! I really like apple pie. I want some! What's the name of the shop?" Let the wife's second speech be: "Hmm, what's it called? I forgot. I remember it's a very long English name."

[0052] In Request Deduction Section 208, the conversation between the wife and her friend is deduced based on the order and content of the speech. Here, based on the wife's statement, "The apple pie shop that recently opened in ZZ Shopping Center is delicious," and the friend's response, "I want to eat it!", it is deduced that the friend wants to eat the apple pie from ZZ Shopping Center. Furthermore, based on this deduction, and the friend's question, "What's the name of the shop?" and the wife's response, "What's it called? I forgot. I remember it's a very long English name," it is deduced that the wife is recalling the background of the name of the apple pie shop in ZZ Shopping Center. Moreover, based on this deduction, it is deduced that the passenger requested to know the name of the apple pie shop in ZZ Shopping Center.

[0053] As another example, consider scenario C below. Let the multiple passengers who spoke be the driver (in the driver's seat) and the passenger (in the front passenger seat). Let the speaking order be the wife, then the driver. Let the wife's speech be, "It's almost noon." Let the driver's speech be, "If it's past noon, all the shops will be packed."

[0054] In the request estimation unit 208, the conversation between the wife and the passenger is inferred based on the order and content of their speech. Here, based on the wife's phrase "It's noon," and the passenger's phrase "If it's past noon, all the shops will be crowded," the background of wanting to quickly find a restaurant is inferred. Furthermore, based on this inference, a passenger's request to eat at a restaurant near their current location is inferred. Additionally, if passenger preference information from the auxiliary information acquired by the auxiliary information acquisition unit 207 is also used, the following can be done. For example, if information such as both the wife and the passenger like ramen is available, a passenger's request to eat at a ramen shop near their current location can be inferred.

[0055] The request estimation unit 208 can also estimate the request of the passenger corresponding to the combination of the passenger states of the multiple passengers in the vehicle as determined by the passenger information determination unit 206. Therefore, even in situations where no conversation occurs, the passenger request can be estimated with higher accuracy. Preferably, the request estimation unit 208 estimates the background of the passenger states of the multiple passengers in the vehicle as determined by the passenger information determination unit 206, and estimates the request of the passenger matching that background. In this case, the background can be estimated based on the combination of the passenger states of the multiple passengers, and the passenger request can be estimated in stages based on the estimated background. Therefore, the passenger request can be estimated with higher accuracy. Furthermore, the passenger information used for estimating the passenger request in the request estimation unit 208 can also be information combining the speech content and the passenger state.

[0056] Here, we will explain an example of how to predict a request from passengers corresponding to a combination of passenger states. Let's assume the following is case D. Let's assume the passengers whose states are determined are the wife in the driver's seat and the infant in the rear seat. Let's assume the infant's passenger state is low alertness. Low alertness means either sleeping or being drowsy to a certain level or higher. The wife's passenger state is turning towards the rear. In request prediction unit 208, based on the combination of these passenger states, we predict whether the wife is concerned about the infant sleeping or in a poor condition. Furthermore, based on this predicted background, we predict the wife's request to clearly confirm the infant's condition.

[0057] As another example, consider scenario E below. Let's assume the occupants whose states are determined are the wife in the driver's seat and the infant in the rear seat. Let's assume the infant's occupant state is low alertness. Let's assume the wife's occupant state is looking towards the rearview mirror and a confused state of mind. In request estimation unit 208, based on the combination of these occupant states, it is estimated that although the wife is concerned about whether the infant is sleeping or in a poor condition, the background is too dark to see clearly in the rearview mirror. Furthermore, based on this estimated background, it is estimated that the wife requests to clearly see the infant's condition.

[0058] As another example, consider the following scenario F. Let's assume the occupant states are determined as the wife in the driver's seat and the infant in the rear seat. Let's assume the infant's occupant state is that they are not sleeping but have low alertness. Let's assume the wife's occupant state is that she is looking towards the rearview mirror and is silent. The silence can be determined by the presence or absence of speech, or by the opening and closing of the mouth in a facial image. In the request estimation unit 208, based on the combination of these occupant states, the background of the wife wanting the infant to sleep is estimated. Furthermore, based on this estimated background, the wife's request to create a car environment conducive to the infant's sleep is estimated. Additionally, if the occupant's preference information from the auxiliary information acquired by the auxiliary information acquisition unit 207 is also used, it can be done as follows. For example, if preference information such as a song frequently played when the infant is sleeping can be acquired, the wife's request to play that song in the car can be estimated.

[0059] The interior environment determination unit 209 determines the interior environment of the vehicle that is presumed to satisfy the occupant's request predicted by the request prediction unit 208. Furthermore, the provision processing unit 203 provides the interior environment determined by the interior environment determination unit 209. In the provision processing unit 203, the interior environment determined by the interior environment determination unit 209 can be provided by providing visual content, auditory content, lighting, in-vehicle air conditioning, fragrance, and dialogue with the in-vehicle AI, either individually or in combination. The provision processing unit 203 can provide visual content via the display device 24. The provision processing unit 203 can provide auditory content and dialogue with the in-vehicle AI via the sound output device 25. The provision processing unit 203 can provide lighting via the lighting device 23. The provision processing unit 203 can provide in-vehicle air conditioning via the air conditioning control ECU 30 and the air conditioning unit 31. The provision processing unit 203 can provide fragrance via the air conditioning control ECU 30 and the fragrance unit 32.

[0060] The interior environment determination unit 209 uses a machine learning learner to infer the interior environment of the vehicle that satisfies the occupant's request, as inferred by the request inference unit 208. In this case, the learner can be a machine learning learner that takes the occupant's request as input and outputs the interior environment that satisfies the request. Alternatively, the interior environment determination unit 209 can also infer the interior environment that satisfies the occupant's request based on the correspondence between the occupant's request and the interior environment that satisfies the request. This correspondence can be obtained by listening to data from multiple subjects.

[0061] The following describes an example of the in-vehicle environment estimated based on a passenger's request estimated by the request estimation unit 208, and an example of providing that in-vehicle environment. For example, in case A above, the process can proceed as follows. In case A, a passenger's request to see fireflies is estimated. In response to this request, the interior environment determination unit 209 estimates the in-vehicle environment for displaying an image of fireflies. Furthermore, the interior environment determination unit 209 can also estimate the in-vehicle environment to provide the atmosphere of when fireflies were seen last year. Moreover, the provision processing unit 203 can display an image of fireflies, or provide the atmosphere of when fireflies were seen last year. Regarding the image of fireflies, if the personal DB205's activity history information includes an image of fireflies taken last year, the provision processing unit 203 can cause the display device 24 to display that image of fireflies. Alternatively, the provision processing unit 203 can also acquire an image of fireflies from the center via the WACM5 and cause the display device 24 to display the acquired image of fireflies. Additionally, if a sound corresponding to the image is also present, the processing unit 203 outputs the sound from the sound output device 25. Regarding the provision of the atmosphere from when the fireflies were seen last year, it can be done as follows: The processing unit 203 controls the air conditioning temperature of the air conditioning unit 31 via the air conditioning control ECU 30 to achieve a room temperature that matches the temperature of the firefly habitat. Furthermore, the processing unit 203 adjusts the brightness of the vehicle interior to match the brightness of the time when the fireflies were seen last year. This dimming is achieved by controlling the lighting device 23 or by controlling the interior lights.

[0062] For example, in scenario B above, the process can be as follows. In scenario B, a passenger requests to know the name of the apple pie shop in ZZ Shopping Center. In response to this request, the indoor environment determination unit 209 determines the indoor environment that will output the sound of an inquiry about the name of the apple pie shop in ZZ Shopping Center. Furthermore, the provision processing unit 203 outputs the sound of an inquiry about the name of the apple pie shop in ZZ Shopping Center. The provision processing unit 203 determines the name of the apple pie shop in ZZ Shopping Center, i.e., XX, by searching the network via WACM5. As an example of sound output, the provision processing unit 203 can output a sound such as "The shop mentioned earlier might be XX" from the sound output device 25 at a timed interval during a conversation between a wife and her friend. Additionally, the provision processing unit 203 can also simultaneously display information about the apple pie shop in ZZ Shopping Center on a display device 24 such as CID. Thus, it can assist passengers in their conversations.

[0063] For example, in scenario C above, the process can be as follows: In scenario C, a passenger's request to dine at a restaurant near their current location is deduced. In response to this request, the indoor environment determination unit 209 deduces the indoor environment in which the sound of a restaurant near the current location is proposed. Furthermore, the provision processing unit 203 outputs the sound of a restaurant near the current location. The provision processing unit 203 determines the location by searching the network for restaurants near the current location via WACM5. As an example of sound output, the provision processing unit 203 can output a sound such as "There is a restaurant called YY 100m ahead" from the sound output device 25 at a timed interval during a conversation between the passenger and their spouse. Additionally, the provision processing unit 203 can also simultaneously display information about the proposed restaurant on a display device 24 such as CID, or display map information indicating the restaurant's location.

[0064] For example, in cases D and E described above, the process can be as follows: In cases D and E, the wife's request to clearly confirm the baby's condition is anticipated. In response to this request, the indoor environment determination unit 209 determines an indoor environment where the display device 24 can easily show the baby's condition, making it convenient for the wife to see. Furthermore, the provision processing unit 203 makes the display device 24 easily visible to the wife show the baby's condition. The provision processing unit 203 displays the image of the baby captured by the indoor camera 21 on display devices such as CID and MID. Additionally, if the occupant information determination unit 206 determines that the baby is sleeping, a low-volume notification to the baby that they are asleep can be output from the sound output device 25 along with the display of the baby's image. This low volume refers to a volume that is assumed not to disturb the baby's sleep.

[0065] For example, in case F above, the process can be as follows. In case F, a wife's request for a car interior environment conducive to the baby's sleep is anticipated. In response to this request, the interior environment determination unit 209 determines an interior environment conducive to the baby's sleep. Furthermore, the provision processing unit 203 provides this environment. If a song frequently played when the baby is sleeping is present, the provision processing unit 203 outputs the song from the sound output device 25 with reduced volume. If no song frequently played when the baby is sleeping is present, the volume of the music output from the sound output device 25 is reduced. If a preferred speaker is used as the sound output device 25, the sound is designed to prevent it from reaching the baby's room area. Additionally, the air conditioning control ECU 30 controls the air conditioning unit 31 to adjust the room temperature and airflow to a level that allows the baby to relax. Temperature adjustments can also be made using a seat heater. The same procedure can be followed when there is a request to put the baby to sleep, or when the request is made by a passenger other than the baby. In this case, the indoor area where the driver is located will not be provided with an indoor environment conducive to falling asleep.

[0066] Furthermore, if the request estimation unit 208 infers the passenger's desired enjoyment, the provision processing unit 203 can provide an indoor environment that is easy for the passenger to enjoy. For example, the provision processing unit 203 can search for music that the passenger likes based on the personal DB205's preference information and output it from the sound output device 25. In this case, it is preferable to ask the passenger for permission to play the music before outputting it, so as not to upset the passenger's mood.

[0067] In addition, when the request estimation unit 208 infers a request to wake the occupant from drowsiness, the provision processing unit 203 can provide an indoor environment that easily wakes the occupant from drowsiness. For example, the provision processing unit 203 can control the air conditioning unit 31 via the air conditioning control ECU 30 to blow out cool air. Alternatively, the provision processing unit 203 can also control the fragrance unit 32 via the air conditioning control ECU 30 to blow out a refreshing fragrance. Furthermore, the provision processing unit 203 can also output fast-paced music from the sound output device 25. Moreover, when the occupant is the driver, the provision processing unit 203 can also output an attention-awakening sound from the sound output device 25.

[0068] <Providing Association Processing in HCU20>

[0069] Next, use Figure 3 The flowchart illustrates an example of a process that provides association processing in HCU20. Figure 3The flowchart can be, for example, a structure that begins with the formal occupant certification of the vehicle using the occupant certification department 201.

[0070] First, in step S1, the processing unit 203 provides an indoor environment for the performance. In step S2, the occupant information determination unit 206 determines the occupant information detected by the sensors used in the vehicle in a manner that distinguishes each occupant of the vehicle.

[0071] In step S3, if auxiliary information exists for the occupant identified in S2 (yes in S3), the process proceeds to step S4. Conversely, if no auxiliary information exists for the occupant identified in S2 (no in S3), the process proceeds to step S5. In step S4, the auxiliary information acquisition unit 207 acquires auxiliary information for the occupant identified in S2, and the process proceeds to step S5.

[0072] In step S5, the request estimation unit 208 estimates the request of the occupant corresponding to the combination of occupant information of multiple occupants of the vehicle, based on the occupant information determined by the occupant information determination unit 206. In step S6, if the occupant's request can be estimated (yes in S6), the process proceeds to step S7. On the other hand, if the occupant's request cannot be estimated (no in S6), the process proceeds to step S9.

[0073] In step S7, the interior environment determination unit 209 determines the interior environment of the vehicle that is presumed to satisfy the occupant's request predicted in S5. In step S8, the provision processing unit 203 provides the interior environment determined in S7.

[0074] In step S9, if the timing for ending the association processing is right (as it is in S9), the association processing ends. Conversely, if the timing for ending the association processing is not right (as it is in S9), the process returns to S2 and repeats. An example of the timing for ending the association processing is when the vehicle's power switch is turned off.

[0075] <Summary of Implementation Method 1>

[0076] According to the structure of Embodiment 1, based on occupant information of multiple occupants of the vehicle determined in a way that distinguishes each occupant of the vehicle, the request of the occupant corresponding to the combination of occupant information of the multiple occupants is inferred. Therefore, it is possible to infer the occupant request that changes according to the relationship and status of the multiple occupants with higher accuracy. In addition, since the occupant information is information about the occupants of the vehicle detected by sensors used in the vehicle, it is possible to infer the occupant request that matches the actual situation with higher accuracy. As a result, by inferring the occupant request in the presence of multiple occupants with higher accuracy, a more comfortable interior experience that matches the occupant request can be provided.

[0077] (Implementation Method 2)

[0078] The above embodiment illustrates a structure for formal occupant authentication of the vehicle using the HCU20, but it is not limited to this. For example, a structure in which an electronic control device other than the HCU20 performs the functions of the occupant authentication unit 201 and the authentication DB202 may also be used.

[0079] (Implementation Method 3)

[0080] In the above embodiments, it is shown that either the speech content or the passenger status can be used as passenger information, but it is not limited to this. For example, a structure can also be adopted in which only either the speech content or the passenger status is used as passenger information.

[0081] (Implementation Method 4)

[0082] In the above embodiment, a structure for using the vehicle system 1 in an automobile is shown, but it is not limited to this. As long as it is a mobile body capable of accommodating multiple occupants, the vehicle system 1 can also be a structure applied to mobile bodies other than automobiles. For example, it can also be a structure used in mobile bodies such as railway vehicles, airplanes, and ships.

[0083] Furthermore, this disclosure is not limited to the embodiments described above. Various modifications can be made within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included within the technical scope of this disclosure. Additionally, the control unit and method described in this disclosure can also be implemented by a dedicated computer, which is configured as a processor programmed to perform one or more functions embodied in a computer program. Alternatively, the apparatus and method described in this disclosure can also be implemented by dedicated hardware logic circuitry. Alternatively, the apparatus and method described in this disclosure can also be implemented by one or more dedicated computers, which are configured as a combination of a processor executing a computer program and one or more hardware logic circuits. Furthermore, the computer program can also be stored as instructions executed by a computer on a non-transferable tangible recording medium that can be read by a computer.

Claims

1. A device for a moving body, which is capable of being used in a moving body, wherein, The mobile body device includes: The occupant information determination unit determines occupant information in a manner that distinguishes each occupant of the mobile body. The occupant information is information about the occupants of the mobile body detected by sensors used in the mobile body. as well as The request estimation unit, using a machine learning learner, estimates the request of the occupant corresponding to a combination of occupant information from multiple occupants of the moving body, based on the occupant information determined by the occupant information determination unit. The learner used performs machine learning by taking (i) a combination of the occupant information of multiple occupants, (ii) the speech content of multiple occupants, and (iii) the order of the speech content as inputs to infer the occupant's request corresponding to the flow of the speech content of the multiple occupants, i.e., the conversation content.

2. The apparatus for mobile bodies according to claim 1, wherein have: An indoor environment determination unit determines the indoor environment of the mobile body that is presumed to satisfy the request of the occupant predicted by the request prediction unit; and A processing unit provides the indoor environment determined by the indoor environment determination unit.

3. The device for a mobile body according to claim 1, wherein, The occupant information determination unit determines, in a manner that distinguishes between individual occupants of the mobile vehicle, at least the content of the occupant's speech detected by the sensor that detects the sound inside the mobile vehicle, as occupant information. The request estimation unit estimates the request of the passenger corresponding to a combination of the speech contents of the multiple passengers of the mobile body, based on the speech contents of the multiple passengers determined by the passenger information determination unit.

4. The device for a mobile body according to claim 3, wherein, The request estimation unit infers the background of the conversation content based on the flow of the speech content of multiple occupants of the mobile body determined by the occupant information determination unit, i.e., the conversation content, and infers the request of the occupant that matches the background.

5. The device for a moving body according to any one of claims 1 to 4, wherein, The occupant information determination unit determines at least one occupant state, either the action or the posture of the occupant detected by the sensors inside the moving body, in a manner that distinguishes each occupant of the moving body, as the occupant information. The request estimation unit estimates the request of the passenger corresponding to a combination of the passenger states of the multiple passengers of the mobile body, based on the passenger states of the multiple passengers determined by the passenger information determination unit.

6. The device for a moving body according to any one of claims 1 to 4, wherein, The system includes an auxiliary information acquisition unit that acquires auxiliary information, which is at least one of the occupant's preferences and past behavior history. The request estimation unit estimates the request based not only on the occupant information of the multiple occupants of the mobile body determined by the occupant information determination unit, but also on the auxiliary information obtained by the auxiliary information acquisition unit.

7. A control method for a moving body, which can be used in the moving body, wherein, The control method for the moving body includes the following steps executed by at least one processor: The occupant information determination process determines occupant information in a manner that distinguishes each occupant of the mobile body. The occupant information is information about the occupants of the mobile body detected by sensors used in the mobile body. as well as The request estimation step uses a machine learning learner to estimate the request for the occupant corresponding to a combination of the occupant information of multiple occupants of the moving body, based on the occupant information determined in the occupant information determination step. The learner used performs machine learning by taking (i) a combination of the occupant information of multiple occupants, (ii) the speech content of multiple occupants, and (iii) the order of the speech content as inputs to infer the occupant request corresponding to the flow of the speech content of the multiple occupants, i.e., the conversation content.

Images