Information processing device, information processing method, and information processing program

Abstract

To prevent a situation that safety of driving is spoiled by state change by an occupant.SOLUTION: A server device 100 comprises: a first detection unit 132 which detects occurrence of predetermined state change to an occupant on the basis of state information showing a state of the occupant of a vehicle; a second detection unit 133 which detects a situation regarding the vehicle; and a storage unit 134 which stores association data in which content information showing a content of the predetermined state change is associated with situation information showing the situation regarding the vehicle when the predetermined state change occurs.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, a mechanism has been proposed for watching over a predetermined person (for example, a child or an elderly person) in a vehicle via a system installed in the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-169942 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-described conventional technology cannot necessarily prevent a situation in which driving safety is compromised due to a change in the state of the occupant.

[0005] For example, in the above-mentioned conventional technology, video of the person being watched over inside the car is recorded, and information extracted from this video is notified to the person watching over the person being watched over.

[0006] As such, the above-mentioned conventional technologies merely assist a person outside the vehicle in properly monitoring those being monitored around the vehicle, and do not take into consideration, for example, ensuring driving safety in terms of changes in the condition of passengers.

[0007] Therefore, the above-mentioned conventional technology cannot necessarily prevent a situation in which driving safety is compromised due to a change in the state of the occupant.

[0008] The present invention has been made in consideration of the above, and aims to realize an information processing device, an information processing method, and an information processing program that can prevent situations in which driving safety is compromised due to changes in the state of the occupants. [Means for solving the problem]

[0009] The information processing device described in claim 1 is characterized by having a first detection unit that detects the occurrence of a predetermined state change for a vehicle occupant based on state information indicating the state of the vehicle occupant, a second detection unit that detects a situation related to the vehicle, and a storage unit that stores linking data that links content information indicating the content of the predetermined state change with situation information indicating the situation related to the vehicle when the predetermined state change occurs.

[0010] The information processing method described in claim 18 is an information processing method executed by an information processing device, and is characterized by including a first detection step of detecting the occurrence of a predetermined state change for a vehicle occupant based on state information indicating the state of the occupant, a second detection step of detecting a situation related to the vehicle, and a storage step of storing linked data linking content information indicating the content of the predetermined state change with situation information indicating the situation related to the vehicle when the predetermined state change occurs.

[0011] The information processing program described in claim 19 is an information processing program for causing an information processing device to execute a first detection procedure for detecting the occurrence of a predetermined state change in a vehicle occupant based on state information indicating the state of the occupant, a second detection procedure for detecting a situation related to the vehicle, and a storage procedure for storing linked data that links content information indicating the content of the predetermined state change with state information indicating the situation related to the vehicle when the predetermined state change occurs. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of an information processing system according to an embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating an overall image of information processing according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of a server device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a sensor information database according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the association information database according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of a server device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a question message information database according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a dialogue realized between the server device and a user. [Figure 9] FIG. 9 is a flowchart showing a detection process procedure for detecting the vehicle situation. [Figure 10] FIG. 10 is a flowchart showing the procedure of the linking process for linking content information with situation information. [Figure 11] FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the server device. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of a form for implementing an information processing device, an information processing method, and an information processing program (hereinafter referred to as an "embodiment") will be described in detail below with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program are not limited to this embodiment. Furthermore, the same components in the following embodiments will be assigned the same reference numerals, and duplicated descriptions will be omitted.

[0014] [1. System Configuration] First, the configuration of an information processing system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of an information processing system according to an embodiment. Fig. 1 shows an information processing system 1 as an example of an information processing system according to an embodiment.

[0015] 1, the information processing system 1 may include an in-vehicle device 10, a user device 60, a server device 100, and a server device 200. The in-vehicle device 10, the user device 60, the server device 100, and the server device 200 are communicably connected via a network N, either wired or wirelessly. The information processing system 1 shown in FIG. 1 may include any number of in-vehicle devices 10, any number of user devices 60, any number of server devices 100, and any number of server devices 200.

[0016] The in-vehicle device 10 may be a dedicated navigation device built into or externally attached to the vehicle VEx, or may be a recording device (drive recorder) installed in the vehicle VEx for crime prevention or to prevent aggressive driving.

[0017] The in-vehicle device 10 may also be configured with a navigation device and a recording device. As an example, the in-vehicle device 10 may be a composite device in which a navigation device and a recording device that are independent from each other are connected to each other so that they can communicate with each other. As another example, the in-vehicle device 10 may be a single device that has both a navigation function and a recording function.

[0018] Furthermore, a user can install a predetermined application into a portable terminal device (for example, a smartphone, tablet terminal, notebook PC, desktop PC, PDA, etc.) that they use on a daily basis, and use the device as the in-vehicle device 10. For example, a portable terminal device on which a predetermined navigation application or a predetermined recording application is installed can be understood as the in-vehicle device 10 referred to here. When the portable terminal device is used as the in-vehicle device 10, it is installed, for example, on the dashboard of the vehicle VEx while driving.

[0019] The in-vehicle device 10 may also include various sensors. For example, the in-vehicle device 10 may include various sensors such as a camera, an acceleration sensor, a gyro sensor, a GPS sensor, and an air pressure sensor. The server device 100, which will be described later, may perform various information processing based on sensor information detected by these sensors. The server device 100 may also use sensor information detected by sensors provided in the vehicle VEx itself as a safe driving system, in addition to the sensors provided in the in-vehicle device 10.

[0020] The user device 60 is a portable terminal device owned by a user. As described above, the portable terminal device may be a smartphone, a tablet terminal, a notebook PC, a desktop PC, a PDA, or the like. In this embodiment, an application AP (hereinafter abbreviated as "app AP") that enables transmission and reception of information between the user device 60 and the server device 100 and the server device 200 is installed in the user device 60. The user can register information with the server device 100 and interact with the interactive function of the server device 200 via the app AP.

[0021] The server device 100 realizes the information processing according to the embodiment by cooperating with the server device 200. For example, the server device 100 detects the occurrence of a predetermined state change in a vehicle occupant based on state information indicating the state of the occupant. The server device 100 may also detect the situation related to the vehicle. As a result, the server device 100 accumulates linking data that links content information indicating the content of the predetermined state change with situation information indicating the situation related to the vehicle when the predetermined state change occurs.

[0022] The server device 200 generates user information for a predetermined user related to the passenger based on the linking data accumulated by the server device 100. When the user to whom the user information has been provided responds to the user information, the server device 200 estimates the cause of the predetermined state change based on the content of the response. The server device 200 then controls so that information according to the estimation result is output to the user.

[0023] Then, according to the information processing performed between the server devices 100 and 200, for example, an in-car abnormality monitoring service (hereinafter, sometimes referred to as "monitoring service SA") is provided to the user. The monitoring service SA enables the user to, for example, look back on the time when their child started crying in the car, find the cause, and take measures to prevent future crying in the car. The user can also receive the monitoring service SA via the application AP. For this reason, the application AP can be said to be an agent (for example, a conversational agent) that provides the monitoring service SA.

[0024] Here, if the in-vehicle device 10 is an edge computer that performs edge processing near the user, the server devices 100 and 200 may be, for example, cloud computers that perform processing on the cloud side.

[0025] Furthermore, without being limited to the example of FIG. 1 , the server device 100 and the server device 200 may be integrated, and the integrated server device corresponds to an information processing device according to the embodiment. Furthermore, the following embodiment shows an example in which information processing according to the embodiment is realized by transmitting and receiving information between the in-vehicle device 10 and the information processing device. However, the information processing according to the embodiment may be realized only on the edge side, i.e., by the in-vehicle device 10. In this case, the in-vehicle device 10 may be configured to behave like an information processing device, for example, by an information processing program according to the embodiment.

[0026] [2. Overview of information processing] From here, an overview of information processing according to the embodiment will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram illustrating an overview of information processing according to the embodiment.

[0027] FIG. 2 shows person U11 and person B12 as passengers of vehicle VE1 (an example of vehicle VEx). More specifically, in the example of FIG. 2, one of the passengers, person U11, is driving the vehicle VE1, and the other passenger, person B12, is sitting in the back seat. Also, in the example of FIG. 2, person U11 and person B12 are in a parent-child relationship, with person U11 being the mother and person B12 being the child. In this regard, person U11 and person B12 can also be said to be in a caregiver-caregiver relationship.

[0028] Here, person U11 is unable to take care of his child, person B12, while driving, and if an incident occurs involving person B12, person U11 may be unable to respond appropriately to the incident, which may compromise driving safety. Such incidents include crying, fussiness, strange vocalizations, or emotional changes. Therefore, person U11 uses monitoring service SA when such incidents occur to investigate the cause and prevent future incidents from occurring.

[0029] 2, it is assumed that person U11 has registered various pieces of information in advance with server device 100 in order to use monitoring service SA. For example, person U11 has registered not only a profile and information for identifying his / her own vehicle, but also information about a person who is a detection target (monitoring target) whose status change is to be detected. According to the example of FIG. 2, person U11 has determined that the person who is a detection target is person B12, and has registered, for example, a facial image of person B12's face and information about the voice of person B12.

[0030] With this registration completed, Figure 2 shows a scene in which person U11 is driving vehicle VE1 with person B12 in the back seat on the morning of October 10, 2021.

[0031] For example, when the vehicle VE1 starts driving, the in-vehicle device 10 may transmit sensor information detected by various sensors provided in the vehicle VE1 to the server device 100 in real time. As described above, some of the sensors may be provided in the in-vehicle device 10 and some may be provided in the vehicle VE1 itself. Therefore, the in-vehicle device 10 can transmit, for example, sensor information detected by sensors inside the vehicle and sensor information detected by sensors outside the vehicle to the server device 100 in real time.

[0032] 2, the server device 100 acquires sensor information from the in-vehicle device 10 (step S11). For example, the server device 100 acquires sensor information detected by a sensor inside the vehicle and sensor information detected by a sensor outside the vehicle.

[0033] Furthermore, each time the server device 100 acquires sensor information, it may detect the situation (vehicle situation) related to the vehicle VE1 at that time based on the acquired sensor information (step S12). For example, the server device 100 may detect, as the situation related to the vehicle VE1, the position of the vehicle VE1, the time at this position, the driving time that the vehicle VE1 has been driving continuously up to now, the in-vehicle environment of the vehicle VE1, or the situation of surrounding vehicles relative to the vehicle VE1. Of course, the information detected by the server device 100 is not limited to the above example.

[0034] The in-vehicle environment of the vehicle VE1 includes the state of the vehicle interior obtained by analyzing the captured image, the sound generated by vocalization and speech, the output sound output from the speakers, the state of vibration, the vibration sound generated by vibration, the engine sound, the temperature inside the vehicle, the humidity inside the vehicle, etc. The conditions of the surrounding vehicles relative to the vehicle VE1 include the output sound leaking out from the speakers of the surrounding vehicles, the engine sounds of the surrounding vehicles, the horns and sirens of the surrounding vehicles, etc.

[0035] In this way, when the server device 100 acquires sensor information and constantly detects the vehicle status of the vehicle VE1 based on the acquired sensor information, it can also determine whether the person B12 to be detected is in the vehicle VE1 based on the detection result. For example, the server device 100 may determine whether the person B12 to be detected is in the vehicle VE1 by comparing the state inside the vehicle (for example, a person's face) or the sound produced by utterances or speech inside the vehicle with the registered information.

[0036] Note that, as in the above example, even if information about the person to be detected (for example, facial image or voice information) is not registered, the server device 100 may determine whether or not a person (young person) of an age group who is likely to experience state changes such as crying, fussing, making strange noises, or emotional changes is riding in the vehicle VE1, based on facial images and voice information detected from sensor information. Then, when the server device 100 determines that such a young person is riding in the vehicle VE1, it may dynamically determine this young person as the detection target.

[0037] In the example of FIG. 2, the server device 100 can determine that the person B12 to be detected is riding in the vehicle VE1, and may perform the following processes in response to this determination.

[0038] For example, when the sensor information includes state information, the server device 100 detects the occurrence of a predetermined state change in person B12 based on the state information (step S13). For example, when the server device 100 can detect vocalization information indicating an utterance by person B12 or speech information indicating an utterance by person B12 from the sensor information, the server device 100 executes a process of detecting whether person B12 has started crying, become fussy, make strange noises, or have a change in emotion based on the detected information (audio information).

[0039] Then, when the server device 100 detects the occurrence of a specified state change, it generates linking data that links content information indicating the content of the detected state change with situation information indicating the vehicle situation of the vehicle VE1 when the specified state change occurs, and stores the generated linking data (step S14).

[0040] For example, when the server device 100 detects that person B12 has started crying based on the status information (audio information) corresponding to person B12, it generates linking data that links content information indicating that person B12 has started crying with situation information of the vehicle VE1 at the time the crying occurred.

[0041] For example, based on the timing at which person B12 starts crying, the server device 100 may generate linking data that links situation information indicating any of the following to the content situation: the position of vehicle VE1 within a specified time range before and after the time at which person B12 starts crying; the in-vehicle environment of vehicle VE1 within this time range; the status of surrounding vehicles relative to vehicle VE1 within this time range; the elapsed time that has passed since person B12 got into vehicle VE1 until he started crying; or the status of driving breaks taken by person B12 between getting into vehicle VE1 and starting to cry.

[0042] Furthermore, the server device 100 may store the association data in an association information database 122 (FIG. 5).

[0043] Up to now, the information processing according to the embodiment has been described using as an example a scene in which person U11 is driving a vehicle VE1 with person B12 in the back seat on the morning of October 10, 2021. From here on, the information processing according to the embodiment will be described using as an example a scene in which person U11, who has finished driving, operates his / her user device 60 at home to use the monitoring service SA to find out why person B12 started crying during the morning drive.

[0044] For example, suppose that the person U11 launches the application AP and performs an operation to call a dialogue agent. In this case, the user device 60 transmits a dialogue request to the server device 200 (step S21). The operation to call a dialogue agent may be, for example, entering a dedicated dialogue room.

[0045] When the server device 200 receives the interaction request, it accesses the server device 100 and refers to the linked data corresponding to the requesting user, i.e., person U11 (step S22). For example, the server device 200 identifies the person to be detected corresponding to person U11 based on the registration information registered by person U11, and refers to the linked data corresponding to the identified person. In the example of Fig. 2, the server device 200 refers to the linked data corresponding to person B12.

[0046] Then, the server device 200 generates a question for estimating the cause of the predetermined change in state of the person B12 based on the linked data (step S23). In the example of Fig. 2, the server device 200 may identify possible causes of the person B12 starting to cry based on the linked data, and generate a question for prompting the person B12 to answer either Yes or No to the identified possible causes.

[0047] Then, the server device 200 starts a dialogue with the person B12 using the generated question (step S24). For example, the server device 200 may cause the user device 60 to display a chat screen C1 virtually showing a dialogue room, and may also cause the user device 60 to display the question in a manner that an agent is speaking to the person B12.

[0048] Here, it is assumed that person U11 inputs an answer to the question provided by server device 200 (for example, selects either Yes or No). In this case, user device 60 transmits answer information indicating the input answer to server device 200 (step S25). An example of a dialogue using a question will be described in detail with reference to FIG. 8.

[0049] Next, the server device 200 estimates the cause of the predetermined change in the state of the person B12 based on the answer input by the person U11 (step S26). In the example of Fig. 2, the server device 200 estimates the fundamental cause of the person B12 starting to cry based on the answer information acquired from the user device 60.

[0050] Furthermore, if the server device 200 can estimate the cause of the predetermined change in the state of person B12, it provides advice information according to the estimation result to person U11 (step S27). In the example of FIG. 2, if the server device 200 can estimate the cause of person B12 starting to cry, it searches for countermeasures to prevent person B12 from starting to cry in the car in the future. Then, the server device 200 outputs advice information indicating the searched countermeasures to the user device 60. For example, the server device 200 may display the advice information on the chat screen C1, or may display the advice information on a predetermined screen corresponding to the application AP.

[0051] An example of information processing according to the embodiment has been described above with reference to FIG. 2. According to such information processing, when a predetermined state change occurs in a person (person B12) to be detected while in a vehicle, the user (person U11) can look back on the time when the state change occurred and find the cause of the state change through dialogue with the agent. Furthermore, the user can appropriately determine what measures should be taken to prevent future state changes from occurring in the vehicle. Furthermore, as a result, the user can prevent a situation in which driving safety is compromised due to a state change caused by the person to be detected.

[0052] [3. Server Device Configuration (1)] From here, the server device 100 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the server device 100 according to the embodiment. As shown in Fig. 3, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0053] (Regarding the communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information to and from, for example, the in-vehicle device 10 and the server device 200.

[0054] (Regarding the storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk, an optical disk, etc. The storage unit 120 has a sensor information database 121 and an association information database 122.

[0055] (About the sensor information database 121) The sensor information database 121 stores sensor information detected by a sensor included in the in-vehicle device 10 or a sensor included in the vehicle VEx itself. An example of the sensor information database 121 according to the embodiment is shown in Fig. 4. In the example of Fig. 4, the sensor information database 121 has items such as "vehicle ID," "detection date and time," "sensor information," and "situation information."

[0056] "Vehicle ID" indicates identification information that identifies the vehicle that sent the "sensor information." "Detection date and time" indicates information regarding the date and time when the "sensor information" was detected. "Sensor information" indicates information detected by a sensor possessed by the vehicle VEx indicated by the "vehicle ID."

[0057] The "status information" is information indicating the status (vehicle status) of the vehicle VEx indicated by the "vehicle ID." The "status information" may be, for example, information detected based on the "sensor information," or may be the "sensor information" itself.

[0058] Furthermore, the "situation information" may include, for example, location information indicating the location of the vehicle VEx indicated by the "vehicle ID," time information indicating the time at this location, time information indicating the continuous driving time of the vehicle VEx indicated by the "vehicle ID," environmental information indicating the in-vehicle environment of the vehicle VEx indicated by the "vehicle ID," surrounding information indicating the status of surrounding vehicles relative to the vehicle VEx indicated by the "vehicle ID," etc.

[0059] In addition, environmental information includes facial images of people inside the vehicle obtained by analyzing captured images (sensor information) of the interior of the vehicle VEx indicated by the ``vehicle ID,'' audio information indicating vocalizations or speech by occupants of the vehicle VEx indicated by the ``vehicle ID,'' output information indicating the output sound output from the speakers of the vehicle VEx indicated by the ``vehicle ID,'' vibration information indicating the vibration of the vehicle VEx indicated by the ``vehicle ID,'' vibration sound information indicating the vibration sound caused by this vibration, engine sound information indicating the engine sound of the vehicle VEx indicated by the ``vehicle ID,'' temperature information indicating the interior temperature of the vehicle VEx indicated by the ``vehicle ID,'' and humidity information indicating the interior humidity of the vehicle VEx indicated by the ``vehicle ID.''

[0060] In addition, the surrounding information includes output information indicating the output sound leaking from the speakers of surrounding vehicles relative to the vehicle VEx indicated by the "vehicle ID," engine sound information indicating the engine sound of surrounding vehicles relative to the vehicle VEx indicated by the "vehicle ID," horn information of surrounding vehicles relative to the vehicle VEx indicated by the "vehicle ID," etc.

[0061] 4 shows an example in which a vehicle ID "VE1", a detection date and time "TM11", sensor information "SC11", and situation information "ST11" are associated with each other. This example shows an example in which a sensor possessed by the vehicle VE1 (for example, a sensor possessed by the in-vehicle device 10 provided in the vehicle VE1, or a sensor possessed by the vehicle VE1 itself) detected the sensor information SC11 at the date and time TM11. This example also shows an example in which situation information ST11 was detected based on the sensor information SC11.

[0062] (About the linking information database 122) The linking information database 122 stores linking data linking content information with status information. Fig. 5 shows an example of the linking information database 122 according to the embodiment. In the example of Fig. 5, the linking information database 122 has items such as "user information," "vehicle ID," "target person information," "occurrence date and time," "content information," "status information," and "situation information." The "situation information" also includes items such as "vehicle position before and after the occurrence," "vehicle environment before and after the occurrence," "surrounding conditions before and after the occurrence," "elapsed time until the occurrence," and "rest conditions before the occurrence."

[0063] "User information" refers to information about a user who has registered information to receive the monitoring service SA. "User information" may include, for example, identification information, a profile, etc. that identifies the user who has registered information to receive the monitoring service SA.

[0064] The "vehicle ID" is an example of registration information, and indicates, for example, identification information that identifies a vehicle driven by a user indicated by the "user information."

[0065] "Target person information" is an example of registration information, and indicates identification information that identifies the person who is the detection target (monitoring target) whose status change is detected. Furthermore, "target person information" may include, for example, a facial image of the person who is the detection target, or information about the voice of the person who is the detection target.

[0066] "Occurrence date and time" indicates information about the date and time when a predetermined state change occurred. "Content information" indicates the content of the predetermined state change that occurred at the "occurrence date and time." In FIG. 5, examples of the content of the predetermined state change include crying, making strange noises, and emotional changes, but other examples include becoming fussy.

[0067] The "status information" is information from among the status information detected from the sensor information that is the basis for detecting a status change indicated by the "content information," and is information related to the status of the person to be detected that is indicated by the "target person information." The "status information" may be, for example, speech information that indicates an utterance by the person to be detected that is indicated by the "target person information," or speech information that indicates the content of an utterance by the person to be detected that is indicated by the "target person information." In other words, the "status information" may be speech information from among the speech information related to the person to be detected that is indicated by the "target person information," that is the basis for detecting a status change.

[0068] As explained above, the state information detected from the sensor information can also be considered as a type of situation information in a broad sense.

[0069] Of course, the "status information" is not limited to the above example, and may be, for example, a facial image showing a crying face, a facial image showing a displeased face, or the like.

[0070] The "status information" is status information that indicates the vehicle status of the vehicle VEx when the status change indicated by the "content information" occurs.

[0071] "Vehicle position before and after occurrence" is information indicating the position of the vehicle VEx within a predetermined time range before and after the time point at which the state change indicated by the "content information" occurred.

[0072] The "surrounding conditions before and after the occurrence" is information indicating the interior environment of the vehicle VEx within a predetermined time range before and after the time when the state change indicated by the "content information" occurred.

[0073] "Time elapsed until occurrence" is information indicating the time elapsed from when the person to be detected indicated by "Target person information" gets into the vehicle VEx until the state change indicated by "Content information" occurs.

[0074] "Rest status up to the time of occurrence" is information indicating the driving rest status between the time the person to be detected indicated by "target person information" gets into the vehicle VEx and the time the status change indicated by "content information" occurs.

[0075] (Regarding the control unit 130) 3, the control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs (for example, the information processing program according to the embodiment) stored in a storage device inside the server device 100 using a RAM as a working area. The control unit 130 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0076] As shown in Fig. 3, the control unit 130 has an acquisition unit 131, a first detection unit 132, a second detection unit 133, a storage unit 134, a generation unit 135, and a prediction unit 136, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 3, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationship between the processing units included in the control unit 130 is not limited to the connection relationship shown in Fig. 3, and may be other connection relationships.

[0077] (Regarding the acquisition unit 131) The acquisition unit 131 acquires sensor information detected by a sensor included in the vehicle VEx (for example, a sensor included in the in-vehicle device 10 provided in the vehicle VEx, or a sensor included in the vehicle VEx itself). Then, the acquisition unit 131 stores the acquired sensor information in the sensor information database 121.

[0078] (Regarding the first detection unit 132) The first detection unit 132 detects the occurrence of a predetermined state change in the occupant of the vehicle VEx based on state information indicating the state of the occupant. For example, if the sensor information acquired by the acquisition unit 131 includes information about a person (for example, vocal information, speech information, or a facial image), the first detection unit 132 acquires this information as state information indicating the state of the occupant. Then, the first detection unit 132 performs a process of detecting the occurrence of a predetermined state change in the occupant based on the acquired state information. For example, the first detection unit 132 detects, as the predetermined state change, the occupant starting to cry, uttering a strange voice, or a change in the occupant's emotions based on the state information.

[0079] (Regarding the second detection unit 133) The second detection unit 133 detects a situation related to the vehicle VEx. For example, the second detection unit 133 may detect the position of the vehicle VEx, the time at this position, the traveling time that the vehicle VEx has been traveling continuously up to now, the interior environment of the vehicle VEx, or the situation of surrounding vehicles relative to the vehicle VEx, based on the sensor information acquired by the acquisition unit 131. Furthermore, the second detection unit 133 stores situation information indicating the detected vehicle situation in the sensor information database 121.

[0080] (Regarding the storage unit 134) The storage unit 134 generates linking data that links content information indicating the content of the predetermined state change with situation information indicating the situation related to the vehicle VEx when the predetermined state change occurs. Then, the storage unit 134 stores the generated linking data in the linking information database 122.

[0081] For example, based on the timing at which a predetermined state change occurs, the accumulation unit 134 may generate linked data that links, to content information, situation information that indicates any of the situations detected by the second detection unit 133, such as the position of the vehicle VEx within a predetermined time range relative to the time at which the predetermined state change occurs, the in-vehicle environment of the vehicle VEx within this time range, the status of surrounding vehicles relative to the vehicle VEx within this time range, the elapsed time that has elapsed since the passenger boarded the vehicle VEx until the predetermined state change occurred, or the status of the passenger taking a driving break between the time at which the passenger boarded the vehicle VEx and the time at which the predetermined state change occurred.

[0082] Furthermore, the storage unit 134 may store state information detected when a predetermined state change occurs among state information detected from the sensor information in a state in which the state information is further associated with the linked data. As a result, the linked information database 122 shown in FIG. 5 is obtained.

[0083] (Regarding the generation unit 135) The generation unit 135 generates a predictive model that predicts whether a predetermined state change will occur for a passenger when situation information indicating the situation regarding the vehicle VEx currently in which the passenger is riding is input, by having the model learn the relationship between the content information contained in the linked data and the situation information.

[0084] (Regarding the prediction unit 136) The prediction unit 136 predicts whether a predetermined state change will occur in the passenger to be processed, based on a trained model that has learned the relationship between the content information and the state information included in the linked data, and the state information that indicates the state of the vehicle that the passenger to be processed is currently riding in. For example, the prediction unit 136 predicts whether a predetermined state change will occur in the passenger to be processed, using the prediction model generated by the generation unit 135 as the trained model.

[0085] For example, the passenger to be processed (the passenger for whom situation information indicating the situation regarding the vehicle VE1 in which the person U11 is currently riding is input) is person B12 in Fig. 2. In this case, the prediction unit 136 can predict whether person B12 will start crying by inputting situation information detected based on sensor information acquired in real time while person U11 is driving the vehicle VE1 into a prediction model. Furthermore, if the prediction unit 136 predicts that person B12 will start crying, it may cause, for example, the in-vehicle device 10 to output information indicating that there is a high possibility that person B12 will start crying if the driving situation remains the same.

[0086] Furthermore, the prediction unit 136 may predict that there is a high possibility that the person B12 will burst into tears, and may also predict the cause of this.

[0087] [4. Server Device Configuration (2)] Next, the server device 200 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the configuration of the server device 200 according to the embodiment. As shown in Fig. 6, the server device 200 includes a communication unit 210, a storage unit 220, and a control unit 230.

[0088] (Regarding the communication unit 860) The communication unit 210 is realized by, for example, a NIC etc. The communication unit 210 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the user device 60 and the server device 100, for example.

[0089] (Regarding the storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk, an optical disk, etc. The storage unit 120 has a question sentence information database 221 and a conversation information database 222.

[0090] (About Question Sentence Information Database 221) The question message information database 221 stores information about questions used in dialogue with users. Here, Fig. 7 shows an example of the question message information database 221 according to the embodiment. In the example of Fig. 7, the question message information database 221 has items such as "Question ID," "Question candidate," and "Target age." Furthermore, the "Target age" includes items such as "Under 1 year old (infant)," "1 to 3 years old (preschooler)," and "3 to 6 years old (kindergartener)."

[0091] "Question ID" indicates identification information that identifies a question candidate.

[0092] The "candidate question" is a candidate question to be provided to a user who has requested a dialogue. The question indicated by the "candidate question" may include, for example, content for obtaining from the requesting user an answer for estimating the cause of a predetermined state change in the person to be detected. The question indicated by the "candidate question" may also include content for making the requesting user reflect on the time when a predetermined state change occurred in the person to be detected.

[0093] "Target age" is condition information that conditions which question sentence should be selected from the questions indicated by the "question sentence candidates" as the question sentence to be provided to the user, from the perspective of the maturity level of the person being detected.

[0094] For example, in the example of FIG. 7, a "Yes" is entered between "under 1 year old (toddler)" and the question "Is it because he wanted milk?". On the other hand, a "Yes" is not entered between "1 to 3 years old (preschooler)" and the question "Is it because he wanted milk?". Also, a "Yes" is not entered between "3 to 6 years old (kindergartener)" and the question "Is it because he wanted milk?".

[0095] In this example, if the age of the person to be detected (person B12 in the example of Figure 2) corresponding to the requesting user (person U11 in the example of Figure 2) is between "1 and 3 years old", the question "Is it because you wanted milk?" is provided to the user, but if the age of the person to be detected is between "1 and 3 years old" or "3 to 6 years old", the question "Is it because you wanted milk?" is not provided to the user.

[0096] As another example in FIG. 7, a "o" is entered between "1 to 3 years old (preschool child)" and the question "Is it because you got bored of the drive?". Also, a "o" is entered between "3 to 6 years old (kindergarten child)" and the question "Is it because you got bored of the drive?". On the other hand, a "o" is not entered between "under 1 year old (toddler)" and the question "Is it because you got bored of the drive?".

[0097] In this example, if the age of the person to be detected corresponding to the requesting user is between "1 and 3 years old" or "3 to 6 years old", the question "Are you bored of driving?" is provided to the user, but if the age of the person to be detected is between "under 1 year old", the question "Are you bored of driving?" is not provided to the user.

[0098] In this way, the "target age" can be said to be condition information that conditions which of the questions indicated by the "candidate questions" are more suitable to be provided to the user from the perspective of the maturity level of the person being detected.

[0099] (About the dialogue information database 222) The interaction information database 222 stores information about interactions between the requesting user and the requesting user. For example, the interaction information database 222 may store history information about interactions between the requesting user and the requesting user.

[0100] (Regarding the control unit 230) 6, the control unit 230 is realized by a CPU, an MPU, or the like executing various programs (for example, the information processing program according to the embodiment) stored in a storage device inside the server device 200 using a RAM as a work area. The control unit 230 is also realized by an integrated circuit such as an ASIC or an FPGA.

[0101] As shown in Fig. 6, the control unit 230 has a request receiving unit 231, a generating unit 232, an estimating unit 233, and an output control unit 234, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 230 is not limited to the configuration shown in Fig. 6, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationship between the processing units included in the control unit 230 is not limited to the connection relationship shown in Fig. 6, and may be other connection relationships.

[0102] (Regarding the request receiving unit 231) The request receiving unit 231 receives, via a dialogue agent, a dialogue request from a user, which is a request to dialogue with the server device 200. When the request receiving unit 231 receives a dialogue request, it may generate a chat screen as a dedicated dialogue room for realizing the dialogue, and may even perform processing to display the generated chat screen on the user device 60.

[0103] (Regarding the generation unit 232) The generation unit 232 generates user information for a predetermined user related to the passenger based on the linked data. For example, the generation unit 232 generates a question including potential causes as content for obtaining an answer from the user used to estimate the cause of a predetermined state change.

[0104] Furthermore, for example, the generation unit 232 generates a question based on the linked data corresponding to a passenger (e.g., a person to be detected) belonging to the requesting user who requested the provision of user information, the question including content that causes the requesting user to look back on the time when a specified state change occurred.

[0105] More specifically, the generation unit 232 identifies candidates for the cause of the occurrence of a predetermined state change based on the linked data, and generates a question asking whether the identified candidates are true or false.

[0106] Furthermore, the generation unit 232 may generate a question with content that corresponds to the maturity level of the passenger.

[0107] (Regarding the estimation unit 233) When the requesting user who has been provided with the user information responds to the user information, the estimation unit 233 estimates the cause of the predetermined status change based on the content of the response. Specifically, when the requesting user inputs an answer to a question, the estimation unit 233 estimates the cause of the predetermined status change of the passenger belonging to the requesting user based on the answer.

[0108] (Regarding the output control unit 234) The output control unit 234 controls so that information according to the estimation result by the estimation unit 233 is output to the user. Specifically, the output control unit 234 controls so that information according to the estimation result by the estimation unit 233 is output to the requesting user.

[0109] For example, the output control unit 234 controls so that advice information regarding a predetermined state change is output as information according to the estimation result. For example, the output control unit 234 controls so that countermeasures to prevent the predetermined state change from occurring are output as advice information.

[0110] [5. Example of a dialogue using a question] Next, an example of a process in which a question is generated by the generation unit 232 and an example of a dialogue between the requesting user and the server device 200 using the generated question will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of a dialogue between the server device 200 and a user. While an example in which a chat screen is displayed is shown below, the dialogue may be conducted using only voice.

[0111] 8 illustrates an example of a dialogue between the server device 200 and the person U11, using the example of steps S21 to S27 described in FIG. 2. Therefore, in the example of FIG. 8, the person U11 corresponds to the requesting user. Also, in FIG. 8, in response to a dialogue request from the person U11, a chat screen C1 virtually showing a dialogue room is displayed on the user device 60 of the person U11.

[0112] In this state, the generation unit 232 accesses the linking information database 122 of the server device 100 and refers to the linking data corresponding to the requesting user, i.e., person U11. For example, the generation unit 232 identifies a person to be detected corresponding to person U11 based on the registration information registered by person U11, and refers to the linking data corresponding to the identified person. According to the examples described above, the generation unit 232 identifies person B12 as the person to be detected corresponding to person U11, and refers to the linking data corresponding to the identified person.

[0113] According to the example of FIG. 5, the generation unit 232 can determine the person B12 as the person to be detected corresponding to the person U11, based on the "target person information" associated with the user information "U11."

[0114] Here, the generation unit 232 controls the question sentence depending on the maturity level of person B12 and the state change that has occurred in person B12. Therefore, in the following, three patterns are prepared for the combination of the age of person B12 and the state change that has occurred in person B12, and each pattern will be described using Figures 8(a), 8(b), and 8(c).

[0115] First, Fig. 8(a) will be described. In Fig. 8(a), the age of person B12 is "less than one year old," and therefore the maturity status of person B12 corresponds to "infant." Also, in Fig. 8(a), it is assumed that a state change of "started crying" has occurred in person B12 inside vehicle VE1 driven by person U11.

[0116] In this example, the generation unit 232 identifies possible causes for why person B12 has started crying based on the linked data corresponding to person B12, specifically, the "status information" and "situation information" corresponding to person B12, among the linked data stored in the linked information database 122. For example, the generation unit 232 may perform a process of identifying possible causes for why person B12 has started crying based on the "status information," and if no possible causes can be identified, may perform a process of identifying possible causes for why person B12 has started crying based on the "situation information."

[0117] In the example of Figure 8(a), if the "status information" is audio information indicating the utterance of person B12, "Milk!", the generation unit 232 can identify "request for milk" as a possible cause of person B12 starting to cry.

[0118] Furthermore, the generation unit 232 extracts question sentences that satisfy the maturity level of "young child" from among the question sentence candidates registered in the question sentence information database 221. According to the question sentence information database 221 shown in Fig. 7, the generation unit 232 can extract questions Q1, Q2, Q3, Q4, Q6, and Q9 as question sentences that satisfy the maturity level of "young child."

[0119] Then, the generation unit 232 selects, from among the questions Q1, Q2, Q3, Q4, Q6, and Q9, the question Q1, which asks whether the reason why person B12 started crying is because he is "requesting milk," as the question to be provided.

[0120] Then, based on the selected question Q1, the generation unit 232 generates a question Q11 in a Yes / No format, such as "Did you start crying while driving because you wanted milk? Yes / No." In this way, the question Q11 is a question for obtaining from person U11 an answer that will serve as evidence for estimating the reason why person B12 started crying, and also has the effect of making person U11 look back on the time when person B12 started crying.

[0121] Furthermore, the generation unit 232 starts a dialogue with the person U11 by displaying a question Q11 on the chat screen C1 as shown in Fig. 8(a). Here, as shown in Fig. 8(a), it is assumed that the person U11 answers "No!" to the question Q11.

[0122] In this case, the generation unit 232 may further generate an additional question to dig deeper into the reason why person B12 started crying, in response to person U11's answer "No!". For example, the generation unit 232 may select a predetermined question from the remaining questions that satisfy the maturity status "infant," i.e., from questions Q2, Q3, Q4, Q6, and Q9. For example, the generation unit 232 may select a question from questions Q2, Q3, Q4, Q6, and Q9 that is closer to the reason why person B12 started crying, based on the "status information" and "situation information" corresponding to person B12. In the example of FIG. 8(a), it is assumed that the generation unit 232 selected question Q2.

[0123] In this case, the generating unit 232 generates a question Q21 in a Yes / No format, such as "So, is it because the diaper was dirty? Yes / No," based on the selected question Q2.

[0124] Then, the generation unit 232 continues the dialogue by displaying the question Q21 on the chat screen C1 as shown in Fig. 8(a). Here, as shown in Fig. 8(a), it is assumed that the person U11 answers "No!" to the question Q21.

[0125] In this case, the generation unit 232 may generate an additional question to further dig into the reason why person B12 started crying, in response to person U11's answer "No!". For example, the generation unit 232 may select a predetermined question from the remaining questions that satisfy the maturity status "infant," i.e., from questions Q3, Q4, Q6, and Q9. For example, the generation unit 232 may select a question from questions Q3, Q4, Q6, and Q9 that is closer to the reason why person B12 started crying, based on the "status information" and "situation information" corresponding to person B12. In the example of FIG. 8(a), it is assumed that the generation unit 232 selected question Q3.

[0126] In this case, the generation unit 232 generates a question Q31 in a Yes / No format, such as "So, was it because you were surprised by the sudden braking? Yes / No," based on the selected question Q3.

[0127] Then, the generation unit 232 continues the dialogue by displaying the question Q31 on the chat screen C1 as shown in Fig. 8(a). Here, as shown in Fig. 8(a), it is assumed that the person U11 answers "Yes!" to the question Q31.

[0128] In such a case, the estimation unit 233 can estimate that the cause of person B12's crying is "surprise at the sudden braking." Furthermore, the output control unit 234 searches for countermeasures to prevent person B12 from crying in the car in the future, based on the cause of person B12's crying, "surprise at the sudden braking." Then, the output control unit 234 controls output so that advice information according to the search result is output to the user device 60.

[0129] As a result, for example, when person B12 is driving the vehicle, the output control unit 234 can warn person U11 not to brake suddenly and provide advice information to person U11 showing specific measures to avoid braking suddenly.

[0130] Next, Fig. 8(b) will be described. In Fig. 8(b), the age of person B12 is "1 to 3 years old", and therefore the maturity status of person B12 corresponds to "kindergartener". Also, in Fig. 8(b), it is assumed that a state change of "starting to make noise" has occurred in person B12 inside vehicle VE1 driven by person U11.

[0131] In this example, the generation unit 232 identifies potential causes of person B12 making a fuss based on the linked data corresponding to person B12, specifically, the "status information" and "situation information" corresponding to person B12, among the linked data stored in the linked information database 122. For example, the generation unit 232 may perform a process of identifying potential causes of person B12 making a fuss based on the "status information," and if no potential cause can be identified, may perform a process of identifying potential causes of person B12 making a fuss based on the "situation information."

[0132] In the example of Figure 8(b), if the "status information" is audio information indicating the speech content of person B12, "I'm bored!!", the generation unit 232 can identify "long ride" as a possible reason why person B12 started making a fuss.

[0133] Furthermore, the generation unit 232 extracts question sentences that satisfy the maturity level of "kindergartener" from among the question sentence candidates registered in the question sentence information database 221. According to the question sentence information database 221 shown in Fig. 7, the generation unit 232 can extract questions Q3, Q4, Q5, Q6, Q7, Q8, and Q9 as question sentences that satisfy the maturity level of "kindergartener."

[0134] Then, the generation unit 232 selects question Q5, which asks whether the reason person B12 started making a fuss is because of the ``long ride,'' as the question to be provided from among questions Q3, Q4, Q5, Q6, Q7, Q8, and Q9.

[0135] Then, based on the selected question Q5, the generation unit 232 generates a question Q51 in a Yes / No format, such as "Did you start making noise while driving because you got bored? Yes / No." In this way, the question Q51 is a question for obtaining from person U11 an answer that will serve as evidence for estimating the reason why person B12 started making noise, and also has the effect of making person U11 look back on the time when person B12 started making noise.

[0136] Furthermore, the generation unit 232 starts a dialogue with the person U11 by displaying a question Q51 on the chat screen C1 as shown in Fig. 8(b). Here, as shown in Fig. 8(b), it is assumed that the person U11 answers "No!" to the question Q51.

[0137] In this case, the generation unit 232 may further generate an additional question to dig deeper into the reason why person B12 started making a fuss, in response to person U11's answer "No!". For example, the generation unit 232 may select a predetermined question from the remaining questions that satisfy the maturity status "kindergartener," i.e., from questions Q3, Q4, Q6, Q7, Q8, and Q9. For example, the generation unit 232 may select a question from questions Q3, Q4, Q6, Q7, Q8, and Q9 that is closer to the reason why person B12 started making a fuss, based on the "status information" and "situation information" corresponding to person B12. In the example of FIG. 8(b), it is assumed that the generation unit 232 selected question Q7.

[0138] In this case, the generation unit 232 generates a question Q71 in a Yes / No format, such as "So, was it because you wanted to go to the toilet? Yes / No," based on the selected question Q7.

[0139] Then, the generation unit 232 continues the dialogue by displaying the question Q71 on the chat screen C1 as shown in Fig. 8(b). Here, as shown in Fig. 8(b), it is assumed that the person U11 answers "No!" to the question Q71.

[0140] In this case, the generation unit 232 may generate an additional question to further dig into the reason why person B12 started making a fuss, in response to person U11's answer "No!". For example, the generation unit 232 may select a predetermined question from the remaining questions that satisfy the maturity status "kindergartener," i.e., from questions Q3, Q4, Q6, Q8, and Q9. For example, the generation unit 232 may select a question from questions Q3, Q4, Q6, Q8, and Q9 that is closer to the reason why person B12 started making a fuss, based on the "status information" and "situation information" corresponding to person B12. In the example of FIG. 8(b), it is assumed that the generation unit 232 selected question Q8.

[0141] In this case, the generating unit 232 generates a question Q81 in a Yes / No format, such as "So, is it because you're hungry? Yes / No," based on the selected question Q8.

[0142] Then, the generation unit 232 continues the dialogue by displaying the question Q81 on the chat screen C1 as shown in Fig. 8(b). Here, as shown in Fig. 8(b), it is assumed that the person U11 answers "Yes!" to the question Q81.

[0143] In such a case, the estimation unit 233 can estimate that "hunger" is the reason why person B12 started making a fuss. Furthermore, the output control unit 234 searches for countermeasures to prevent person B12 from making a fuss in the future in the car, based on the reason "hunger" that person B12 started making a fuss. Then, the output control unit 234 controls output so that advice information according to the search result is output to the user device 60.

[0144] As a result, for example, when person U11 is driving with person B12 in the car, the output control unit 234 can provide advice information to suggest that person U11 take a break from time to time and give snacks to person B12.

[0145] Next, Fig. 8(c) will be described. In Fig. 8(c), the age of person B12 is "less than one year old", and therefore the maturity status of person B12 corresponds to "infant". Also, in Fig. 8(c), it is assumed that a state change of "started crying" has occurred in person B12 inside vehicle VE1 driven by person U11.

[0146] In this example, the generation unit 232 identifies possible causes for why person B12 has started crying based on the linked data corresponding to person B12, specifically, the "status information" and "situation information" corresponding to person B12, among the linked data stored in the linked information database 122. For example, the generation unit 232 may perform a process of identifying possible causes for why person B12 has started crying based on the "status information," and if no possible causes can be identified, may perform a process of identifying possible causes for why person B12 has started crying based on the "situation information."

[0147] In the example of Figure 8(c), if the "situation information" is surrounding information indicating the siren sound "beep beep" of person B12, the generation unit 232 can identify "surprise at the siren of a nearby vehicle" as a possible cause of person B12 starting to cry.

[0148] Furthermore, the generation unit 232 extracts question sentences that satisfy the maturity level of "young child" from among the question sentence candidates registered in the question sentence information database 221. According to the question sentence information database 221 shown in Fig. 7, the generation unit 232 can extract questions Q1, Q2, Q3, Q4, Q6, and Q9 as question sentences that satisfy the maturity level of "kindergartener."

[0149] Then, the generation unit 232 selects, from among the questions Q1, Q2, Q3, Q4, Q6, and Q9, question Q6, which asks whether the reason person B12 started crying was because he was "surprised by the siren of a nearby vehicle," as the question to be provided.

[0150] Then, based on the selected question Q6, the generation unit 232 generates a question Q61 in a Yes / No format, such as "Did you start crying while driving because you were startled by the siren? Yes / No." In this way, the question Q61 is a question for obtaining from person U11 an answer that will serve as evidence for the reason why person B12 started crying, and also has the effect of making person U11 look back on the time when person B12 started crying.

[0151] Furthermore, the generation unit 232 starts a dialogue with the person U11 by displaying a question Q61 on the chat screen C1 as shown in Fig. 8(c). Here, as shown in Fig. 8(c), it is assumed that the person U11 answers "No!" to the question Q61.

[0152] In this case, the generation unit 232 may further generate an additional question to dig deeper into the reason why person B12 started crying, depending on person U11's answer "No!". For example, the generation unit 232 may select a predetermined question from the remaining questions that satisfy the maturity status "infant," i.e., from questions Q1, Q2, Q3, Q4, and Q9. For example, the generation unit 232 may select a question from questions Q1, Q2, Q3, Q4, and Q9 that is closer to the reason why person B12 started crying, based on the "status information" and "situation information" corresponding to person B12. In the example of FIG. 8(c), it is assumed that the generation unit 232 selected question Q3.

[0153] In this case, the generation unit 232 generates a question Q31 in a Yes / No format, such as "So, was it because you were surprised by the sudden braking? Yes / No," based on the selected question Q3.

[0154] Then, the generation unit 232 continues the dialogue by displaying the question Q31 on the chat screen C1 as shown in Fig. 8(c). Here, as shown in Fig. 8(c), it is assumed that the person U11 answers "No!" to the question Q31.

[0155] In this case, the generation unit 232 may generate an additional question to further dig into the reason why person B12 started crying, in response to person U11's answer "No!". For example, the generation unit 232 may select a predetermined question from the remaining questions that satisfy the maturity status "infant," i.e., from questions Q1, Q2, Q4, and Q9. For example, the generation unit 232 may select a question from questions Q1, Q2, Q4, and Q9 that is closer to the reason why person B12 started crying, based on the "status information" and "situation information" corresponding to person B12. In the example of FIG. 8(c), it is assumed that the generation unit 232 selected question Q4.

[0156] In this case, the generating unit 232 generates a question Q41 in a Yes / No format, such as "So, was it because it was hot? Yes / No," based on the selected question Q4.

[0157] Then, the generation unit 232 continues the dialogue by displaying the question Q41 on the chat screen C1 as shown in Fig. 8(c). Here, as shown in Fig. 8(c), it is assumed that the person U11 answers "Yes!" to the question Q41.

[0158] In such a case, the estimation unit 233 can estimate that the cause of person B12's crying is "interior temperature rise." Furthermore, the output control unit 234 searches for countermeasures to prevent person B12 from making a fuss in the car in the future, based on the cause of person B12's crying, "interior temperature rise." Then, the output control unit 234 controls output so that advice information according to the search result is output to the user device 60.

[0159] As a result, for example, when person B12 gets into the car when the temperature is high, the output control unit 234 can provide person U11 with advice information to warn him or her to be careful about the temperature inside the car and to advise him or her not to leave person B12 inside the car.

[0160] [6. Processing Procedure] Next, the procedure of information processing realized by the server device 100 will be described with reference to Figures 9 and 10. The information processing procedure can be roughly divided into two steps: a detection step of detecting the vehicle status based on sensor information, and a step of linking, when a status change is detected based on status information included in the sensor information, content information indicating the status change with the information on the vehicle status at that time. Therefore, the procedure of the vehicle status detection step will be described with reference to Figure 9, and the procedure of the linking step will be described with reference to Figure 10.

[0161] 9 and 10, information processing by the server device 100 is explained using an example of a scene in which person U11, the mother of person B11, is driving a vehicle VE1, assuming that information about person B12, the person to be detected, is registered in the server device 100.

[0162] [6-1. Processing Procedure (1)] First, the procedure of information processing performed in the vehicle condition detection step will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the procedure of the detection process for detecting the vehicle condition.

[0163] 9, first, the acquisition unit 131 determines whether or not driving of the vehicle VE1 has started (step S901). While the acquisition unit 131 determines that driving of the vehicle VE1 has not started (step S901; No), the acquisition unit 131 waits until it can determine that driving of the vehicle VE1 has started.

[0164] On the other hand, when it is determined that driving of the vehicle VE1 has started (step S901; Yes), the acquisition unit 131 acquires sensor information sent in real time from the in-vehicle device 10 in accordance with the driving of the vehicle VE1 (step S902). In addition, the acquisition unit 131 may store the acquired sensor information in the sensor information database 121.

[0165] Every time the acquisition unit 131 acquires sensor information, the second detection unit 133 detects the vehicle situation regarding the vehicle VEx based on the acquired sensor information (step S903). For example, the second detection unit 133 may detect the position of the vehicle VE1, the time at this position, the traveling time that the vehicle VE1 has been traveling continuously up to now, the in-vehicle environment of the vehicle VE1, or the situation of surrounding vehicles relative to the vehicle VE1. Furthermore, the second detection unit 133 may store situation information indicating the detected vehicle situation in the sensor information database 121.

[0166] Next, the second detection unit 133 determines whether or not the driving of the vehicle VE1 has ended (step S904). For example, the second detection unit 133 may determine whether or not the driving of the vehicle VE1 has ended based on whether or not the sensor information has been received from the acquisition unit 131. For example, the second detection unit 133 determines that the driving of the vehicle VE1 is continuing while the sensor information can be received from the acquisition unit 131 (step S904; No), and executes the processing from step S903 again. On the other hand, when the second detection unit 133 is no longer able to receive sensor information from the acquisition unit 131, it determines that the driving of the vehicle VE1 has ended (step S904; Yes), and ends the processing.

[0167] [6-2. Processing Procedure (2)] Next, the procedure of information processing performed in the linking step will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the procedure of the linking process for linking content information with situation information.

[0168] First, the first detection unit 132 determines whether or not the person B12 to be detected is present among the passengers of the vehicle VE1 based on the sensor information acquired by the acquisition unit 131 (step S1001). For example, when the sensor information includes information about a person (for example, vocal information, speech information, a face image, etc.), the first detection unit 132 may determine whether or not the person B12 to be detected is present among the passengers of the vehicle VE1 by comparing the information about the person with the registered information.

[0169] If the first detection unit 132 determines that the person B12 to be detected is not present among the passengers of the vehicle VE1 (step S1001; No), the first detection unit 132 may end the linking process.

[0170] On the other hand, when the first detection unit 132 determines that the person B12 to be detected is present among the occupants of the vehicle VE1 (step S1001; Yes), it determines whether information about the person is included in the sensor information acquired from time to time by the acquisition unit 131 (step S10002). For example, the first detection unit 132 may determine whether the information about the person includes speech information indicating an utterance by the occupant of the vehicle VE1, speech information indicating the content of an utterance by the occupant of the vehicle VE1, or a facial image of the occupant of the vehicle VE1.

[0171] When the first detecting unit 132 determines that the sensor information does not include information about a person (step S1002; No), the first detecting unit 132 waits until sensor information including information about a person is acquired.

[0172] On the other hand, if the first detection unit 132 determines that the sensor information includes information about a person (step S1002; Yes), it determines whether the information about the person indicates the person B12 to be detected (step S1003).

[0173] If the first detecting unit 132 determines that the information relating to a person is not information indicating the person B12 who is the detection target (step S1003; No), the first detecting unit 132 returns the process to step S1002.

[0174] On the other hand, if the first detection unit 132 determines that the information regarding the person is information indicating the person B12 to be detected (step S1003; Yes), it acquires this information as status information indicating the status of the person B12 to be detected (step S1004).

[0175] Then, the first detection unit 132 performs a process of detecting occurrence of a predetermined state change in the person B12 to be detected based on the acquired state information (step S1005). For example, the first detection unit 132 detects, based on the state information, whether the person to be detected has started crying, whining, making a fuss, uttering strange noises, or showing a change in emotion (for example, a change to a bad mood).

[0176] Next, the storage unit 134 determines whether or not the occurrence of a predetermined state change has been detected by the first detection unit 132 (step S1006). If the storage unit 134 determines that the occurrence of a predetermined state change has not been detected (step S1006; No), the process returns to step S1002.

[0177] On the other hand, if the accumulation unit 134 determines that the occurrence of a specified state change has been detected (step S1006; Yes), it generates linking data that links content information indicating the content of the specified state change with situation information regarding the vehicle VE1 when the specified state change occurred (step S1007).

[0178] For example, based on the timing at which a predetermined state change occurs, the accumulation unit 134 may generate linked data in which situation information indicating any of the vehicle conditions detected by the second detection unit 133, such as the position of vehicle VE1 within a predetermined time range relative to the time at which the predetermined state change occurs, the in-vehicle environment of vehicle VE1 within this time range, the status of surrounding vehicles relative to vehicle VE1 within this time range, the elapsed time since person B12 boarded vehicle VE1 until the predetermined state change occurred, or the status of driving breaks taken by person B12 between the time at which person B12 boarded vehicle VE1 and the time at which the predetermined state change occurred, is linked to content information.

[0179] Furthermore, the storage unit 134 stores the generated linking data in the linking information database 122 (step S1008).

[0180] [7. Modifications] The information processing devices (server device 100, server device 200) according to the above-described embodiments may be implemented in various different forms other than the above-described embodiments. Therefore, other embodiments of the information processing device according to the embodiment will be described below.

[0181] In the above embodiment, an example has been shown in which the estimation unit 233 estimates the cause of a predetermined state change from the user's response to the user information generated based on the linked data. In this way, when the cause of the state change is estimated, the accumulation unit 134 may generate linked data by not only linking content information indicating the content of the state change with status information indicating the state of the vehicle when the state change occurred, but also linking cause information indicating the cause of the occurrence (for example, lack of milk, dirty diaper, long ride, etc.).

[0182] Furthermore, the generation unit 135 may generate a prediction model that not only predicts whether a predetermined state change will occur or not, but also predicts the cause of the occurrence of the predetermined state change, based on the linking data described above.

[0183] In this case, the prediction unit 136 can use the prediction model to predict, for example, the timing at which the person B12 will start crying and the cause of the crying.

[0184] Furthermore, when the prediction unit 136 predicts that the cause of crying is "lack of milk" using a prediction model, the output control unit 234 may provide information suggestion such as "the baby is about to cry, so stop the vehicle and encourage the baby to drink milk." Furthermore, when the prediction unit 136 predicts that the cause of crying is "diaper soiling due to excrement" using a prediction model, the output control unit 234 may provide information suggestion such as "the baby is about to cry, so stop the vehicle and encourage the baby to drink diaper."

[0185] As another example, by using a prediction model, the prediction unit 136 can predict, for example, the timing at which person B12 will start making a fuss and the cause of that fuss.

[0186] Furthermore, when the prediction unit 136 predicts that a "long ride" is the cause of fussiness using a prediction model, the output control unit 234 may provide information suggestions such as "encouraging the passenger to take a break in advance," "encouraging the passenger to watch a video to watch in advance," or "encouraging the passenger to engage in recreational activities that can be practiced in the car (for example, word games or simple games)."

[0187] According to the information processing device of this modified example, measures to prevent an incident occurring to a passenger (e.g., person B12) can be presented to relevant parties (e.g., person U11) and encouraged to implement them, thereby effectively preventing incidents from occurring inside the vehicle.

[0188] [8. Hardware Configuration] The server devices 100 and 200 according to the above-described embodiments are realized, for example, by a computer 1000 configured as shown in Fig. 11. The following description will be given taking the server device 100 as an example. Fig. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of the server device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0189] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0190] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0191] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0192] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0193] For example, when the computer 1000 functions as the server device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0194] Furthermore, when the computer 1000 functions as the server device 200 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200, thereby realizing the functions of the control unit 230. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0195] [9. Other] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0196] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0197] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0198] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have undergone various modifications and improvements based on the knowledge of those skilled in the art. [Explanation of symbols]

[0199] 1. Information Processing Systems 10 Onboard equipment 60 User equipment 100 Server device 120 Storage section 121 Sensor Information Database 122 Linking Information Database 130 control section 131 Acquisition Department 132 First detection unit 133 Second detection unit 134 Storage Unit 135 Generation part 136 Prediction Department 200 Server device 220 Storage section 221 Question Information Database 222 Dialogue Information Database 230 Control Unit 231 Request Reception Department 232 Generation part 233 Estimation Department 234 Output control section

Claims

1. a first detection unit that detects occurrence of a predetermined state change in a person to be detected who is a passenger of a vehicle, based on state information indicating the state of the person to be detected; a second detection unit that detects a situation related to the vehicle; a generation unit that generates question information for a user of the vehicle, the user being different from the person to be detected, based on content information indicating the content of the predetermined state change, situation information indicating the situation related to the vehicle when the predetermined state change occurred, and information regarding the maturity state of the person to be detected; an estimation unit that estimates a cause of the occurrence of the predetermined state change in the detection target person based on a response by the user to the question information; an output control unit that controls information according to the estimation result by the estimation unit to be output to the user; An information processing device comprising:

2. A storage unit that stores linked data that links content information indicating the content of the predetermined state change with situation information that indicates the situation regarding the vehicle when the predetermined state change occurs, The generation unit generates the question information based on the linked data and information on the maturity status of the detection subject.

2. The information processing apparatus according to claim 1, wherein:

3. The generating unit generates, as the question information, a question sentence including the candidate causes as content for obtaining an answer used to estimate a cause of the occurrence of the predetermined state change.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The output control unit controls so that advice information regarding the predetermined state change is output as information according to the estimation result.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The output control unit controls so that a countermeasure for preventing the predetermined state change from occurring is output as the advice information.

5. The information processing apparatus according to claim 4,

6. The vehicle further includes a prediction unit that predicts whether the predetermined state change will occur in the passenger to be processed based on the linked data.

3. The information processing apparatus according to claim 2, wherein:

7. The prediction unit predicts whether the predetermined state change will occur in the passenger to be processed, based on a trained model that has learned the relationship between the content information and the situation information included in the linked data, and situation information that indicates a situation related to a vehicle in which the passenger to be processed is currently riding.

7. The information processing apparatus according to claim 6,

8. An information processing method executed by an information processing device, a first detection step of detecting occurrence of a predetermined state change in a person to be detected, based on state information indicating a state of the person to be detected who is a passenger of a vehicle; a second detecting step of detecting a situation related to the vehicle; a generation step of generating question information for a user of the vehicle, who is different from the person to be detected, based on content information indicating the content of the predetermined state change, situation information indicating the situation related to the vehicle when the predetermined state change occurred, and information regarding the maturity state of the person to be detected; an estimation step of estimating a cause of the occurrence of the predetermined state change in the detection target person based on a response by the user to the question information; an output control step of controlling so that information corresponding to the estimation result in the estimation step is output to the user; An information processing method comprising:

9. a first detection step of detecting occurrence of a predetermined state change in a person to be detected, based on state information indicating a state of the person to be detected who is a passenger of a vehicle; a second detection step for detecting a situation related to the vehicle; a generation step of generating question information for a user of the vehicle, who is different from the person to be detected, based on content information indicating the content of the predetermined state change, situation information indicating the situation related to the vehicle when the predetermined state change occurs, and information regarding the maturity state of the person to be detected; an estimation step of estimating a cause of the occurrence of the predetermined state change in the detection target person based on a response by the user to the question information; an output control procedure for controlling information corresponding to an estimation result obtained by the estimation procedure to be output to the user; An information processing program for causing an information processing device to execute the above.

Images