Information processing device, information processing system, information processing method, and recording medium

Abstract

This information processing device comprises a first image processing unit, a first generating unit, and a determining unit. The first image processing unit performs first image processing on a driver image obtained by imaging a driver who is driving a vehicle, and generates motion data related to motions of the driver while driving. On the basis of the history of the motion data, the first generating unit generates first individual standard data relating to standard motions performed by the driver while driving the vehicle. The determining unit determines whether or not a relationship between the motion data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies an output condition for outputting predetermined output information.

Description

Information processing device, information processing system, information processing method, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a recording medium.

[0002] For example, Patent Document 1 discloses a system for determining the degree of driver concentration on driving. The system described in Patent Document 1 determines the degree of driver concentration on driving using signals applied from a steering angle sensor that detects the displacement of a steering wheel and a front camera sensor that detects the road ahead of the vehicle and objects within a predetermined area.

[0003] JP 2013-89233 A

[0004] However, Patent Document 1 does not disclose any technology for assisting safe driving using sensors other than the steering angle sensor and the front camera sensor.

[0005] One of the objectives of the present disclosure is to support safe driving.

[0006] The information processing device of the present disclosure comprises: a first image processing means that performs first image processing on a driver image captured of a driver while driving a vehicle to generate behavior data related to the behavior of the driver while driving; a first generation means that generates first individual standard data related to standard behavior performed by the driver while driving the vehicle based on a history of the behavior data; and a judgment means that judges whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies an output condition for outputting predetermined output information.

[0007] The information processing system of the present disclosure comprises an information processing device and an in-vehicle terminal mounted on a vehicle, wherein the information processing device comprises: a first image processing means that performs first image processing on a driver image captured of the driver while driving the vehicle to generate behavior data related to the behavior of the driver while driving; a first generation means that generates first individual standard data related to standard behavior performed by the driver while driving the vehicle based on a history of the behavior data; and a judgment means that judges whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies an output condition for outputting predetermined output information, and the in-vehicle terminal outputs the output information based on the judgment result of whether the output condition is satisfied.

[0008] The information processing method of the present disclosure includes one or more computers performing first image processing on a driver image captured of a driver driving a vehicle to generate behavior data regarding the behavior of the driver while driving, generating first individual standard data regarding standard behavior performed by the driver while driving the vehicle based on a history of the behavior data, and determining whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies output conditions for outputting predetermined output information.

[0009] The program in the present disclosure is a program that causes one or more computers to perform the following: perform first image processing on a driver image captured of a driver while driving a vehicle to generate behavior data regarding the behavior of the driver while driving; generate first individual standard data regarding standard behavior performed by the driver while driving the vehicle based on a history of the behavior data; and determine whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies output conditions for outputting predetermined output information.

[0010] According to the present disclosure, it is possible to support safe driving.

[0011] 1 is a block diagram showing a configuration example of a first information processing system in the present disclosure. FIG. 1 is a block diagram showing a configuration example of a first information processing device in the present disclosure. FIG. 2 is a flowchart showing processing operations of a first information processing device in the present disclosure. FIG. 3 is a block diagram showing a configuration example of a second information processing system in the present disclosure. FIG. 4 is a block diagram showing a configuration example of a second information processing device in the present disclosure. FIG. 5 is a flowchart showing processing operations of a second information processing device in the present disclosure. FIG. 6 is a flowchart showing processing operations of a second information processing device in the present disclosure. FIG. 7 is a block diagram showing a configuration example of a third information processing system in the present disclosure. FIG. 8 is a block diagram showing a configuration example of a third information processing device in the present disclosure. FIG. 9 is a flowchart showing processing operations of a third information processing device in the present disclosure. FIG. 10 is a flowchart showing processing operations of a third information processing device in the present disclosure. FIG. 11 is a block diagram showing a configuration example of a fourth information processing device in the present disclosure. FIG. 12 is a flowchart showing processing operations of a fourth information processing device in the present disclosure. FIG. 13 is a flowchart showing processing operations of a fourth information processing device in the present disclosure. FIG. 14 is a flowchart showing processing operations of a fourth information processing device in the present disclosure. FIG. 15 is a block diagram showing a physical configuration example of an information processing device in the present disclosure.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all drawings, similar components are denoted by similar reference numerals, and descriptions thereof will be omitted as appropriate. In the present disclosure, the drawings relate to one or more embodiments.

[0013] First Embodiment An information processing system S1 includes an information processing device 100 and an in-vehicle device P mounted on a vehicle C, as shown in FIG.

[0014] (Configuration Example of Information Processing Apparatus 100) As shown in FIG. 2, the information processing apparatus 100 includes a first image processing unit 110, a first generating unit 120, and a determining unit 130, for example.

[0015] The first image processing unit 110 performs first image processing on a driver image captured of the driver while driving a vehicle, and generates motion data relating to the motion of the driver while driving.

[0016] The first generating unit 120 generates first individual standard data relating to standard actions performed by a driver while driving a vehicle, based on the history of the action data.

[0017] The judgment unit 130 judges whether the relationship between the operation data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies the output conditions for outputting predetermined output information.

[0018] The in-vehicle device P outputs output information based on the result of the determination as to whether the output condition is satisfied.

[0019] (Example of Processing Operation of Information Processing Device 100) The information processing device 100 executes information processing as shown in FIG. 3, for example.

[0020] The first image processing unit 110 performs first image processing on a driver image captured of the driver while driving the vehicle, and generates motion data relating to the motion of the driver while driving (step S110).

[0021] The first generating unit 120 generates first individual standard data relating to standard actions performed by a driver while driving a vehicle, based on the history of the action data (step S120).

[0022] The judgment unit 130 judges whether the relationship between the operation data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies the output conditions for outputting predetermined output information (step S130).

[0023] As described above, the information processing device 100 of the present embodiment can appropriately determine whether to output output information based on the driver while driving a vehicle, using a driver image captured of the driver while driving the vehicle. Therefore, it is possible to easily support safe driving.

[0024] According to the information processing system S1 of this embodiment, it is possible to use a driver image captured of the driver while driving a vehicle to output appropriate output information according to the driver while driving, thereby easily supporting safe driving.

[0025] According to the information processing of this embodiment, it is possible to appropriately determine whether to output output information based on the driver who is driving the vehicle, using a driver image captured of the driver while driving the vehicle, thereby easily supporting safe driving.

[0026] [Embodiment 2] In this embodiment, a detailed configuration example of the information processing system described in embodiment 1 will be described, taking as an example a case where a driver who is driving a vehicle C is identified from among a plurality of drivers and the identified driver is supported in driving the vehicle C. Note that in this embodiment, for the sake of simplicity, descriptions that overlap with other embodiments will be omitted as appropriate.

[0027] (Configuration Example of Information Processing System S2) The information processing system S2 includes, for example, an in-vehicle camera CI and an in-vehicle device P mounted on a vehicle C, and an information processing device 200, as shown in FIG.

[0028] The information processing device 200 and devices, equipment, etc. mounted on the vehicle C (in-vehicle camera CI and in-vehicle device P in this embodiment) are connected to each other via, for example, a communication network NT, and may transmit and receive information to and from each other via the communication network NT. The communication network NT is, for example, a wired network, a wireless network, or a network configured by a combination of these.

[0029] Note that some or all of the functions of the in-vehicle device P described below may be provided in the information processing device 200. Furthermore, the in-vehicle device P may serve as an information processing device and include some or all of the functions of the information processing device 200 described below. Furthermore, various devices such as an in-vehicle camera CI mounted on the vehicle C may be provided in the in-vehicle device P.

[0030] (Regarding the in-vehicle camera CI) The in-vehicle camera CI is an example of an imaging device that captures images of the interior of a vehicle. Here, the interior of the vehicle means the inside of the vehicle C. The in-vehicle camera CI is installed in the vehicle so that the imaging range includes the driver's seat, for example, and captures images of the driver who is driving the vehicle C. Note that the vehicle C may be equipped with multiple in-vehicle cameras CI.

[0031] Here, "during driving" typically refers to a period during which the vehicle C is in operation, such as when the vehicle C is traveling, but is not limited to this. For example, "during driving" may refer to a period during which the driver is sitting in the driver's seat.

[0032] The in-vehicle camera CI transmits, for example, a driver image captured of the driver while driving the vehicle C to the information processing device 200. For example, the in-vehicle camera CI may continuously capture images of the driver while driving the vehicle C and transmit the captured driver images to the information processing device 200 in real time.

[0033] The in-vehicle camera CI may be implemented using various types of imaging devices, such as a camera that generates monochrome images or a camera that generates color images.

[0034] (Regarding the in-vehicle device P) The in-vehicle device P is a device mounted on the vehicle C. The in-vehicle device P may be, for example, a car navigation system or the like, but is not limited thereto. The in-vehicle device P may be, for example, a smartphone, a tablet terminal, or the like carried by the driver.

[0035] As described above, the in-vehicle device P outputs predetermined output information based on the determination result of whether the output condition is satisfied. For example, when the in-vehicle device P obtains the determination result of whether the output condition is satisfied from the determination unit 130, the in-vehicle device P outputs predetermined output information based on the determination result. Examples of when the output information is output and examples of the output information will be described later.

[0036] (Regarding the information processing device 200) The information processing device 200 is a device that assists a driver in driving a vehicle C. As shown in FIG. 5 , the information processing device 200 includes a driver identification unit 210, the first image processing unit 110, the first generation unit 120, and the determination unit 130.

[0037] The driver identification unit 210 performs a first image processing on a driver image captured while the driver is driving a vehicle, and generates behavior data relating to the behavior of the driver while driving.

[0038] The driver identification unit 210 identifies the driver who is driving.

[0039] (Example of Processing Operation of Information Processing Device 200) The information processing device 200 executes, for example, information processing as shown in Fig. 6. This information processing is an example of processing for generating first individual standard data.

[0040] The first image processing unit 110 performs a first image processing on a driver image captured while the driver is driving a vehicle, to generate operation data related to the operation of the driver while driving (step S110a). In step S110a, for example, the first image processing unit 110 may perform the first image processing on a history of driver images captured during a predetermined first time length, to generate a history of operation data related to the operation of the driver while driving.

[0041] The first generating unit 120 generates first individual standard data relating to standard actions performed by a driver while driving a vehicle, based on the history of the action data generated in step S110a (step S120).

[0042] The information processing device 200 also executes information processing such as that shown in Fig. 7. This information processing is an example of processing for determining whether or not an output condition is satisfied.

[0043] The driver identification unit 210 identifies the driver who is driving (step S210).

[0044] The first image processing unit 110 performs first image processing on a driver image captured of the driver while driving the vehicle to generate motion data related to the motion of the driver while driving (step S110b). In step S110b, for example, the first image processing unit 110 may perform first image processing on a driver image captured of the driver while driving the vehicle in real time to generate motion data related to the motion of the driver while driving.

[0045] The judgment unit 130 judges whether the relationship between the operation data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies the output conditions for outputting predetermined output information (step S130).

[0046] (Driver Identification Unit 210) As described above, the driver identification unit 210 identifies the driver who is driving. That is, the driver who is driving may be, for example, a driver identified by the driver identification unit 210.

[0047] For example, the driver identification unit 210 uses biometric authentication to identify the driver currently driving the vehicle C from among a plurality of drivers registered in advance.

[0048] For example, a driver image may be used for the biometric authentication. For example, the driver identification unit 210 may acquire a driver image from the in-vehicle camera CI at a predetermined timing, such as when the driver sits in the driver's seat of the vehicle C or when the vehicle C starts operating.

[0049] The fact that the driver is sitting in the driver's seat may be detected, for example, by using a sensor that detects a person in the driver's seat. This sensor may be constantly operating even while the vehicle C is stopped. The fact that the vehicle C has started to operate may be, for example, when the driver issues a predetermined command to the vehicle C to start the vehicle C.

[0050] For example, the driver identification unit 210 may detect the driver's face included in the driver image and perform facial authentication using the detected driver's face (facial image). Facial authentication is a type of biometric authentication using a facial image, and may be performed by, for example, comparing feature amounts extracted from the facial image with pre-registered feature amounts. Note that the facial authentication is not limited to the example given here, and general techniques may be used.

[0051] Although facial authentication has been described as an example here, the biometric authentication used by the driver identification unit 210 is not limited to facial authentication, and for example, fingerprint authentication, voiceprint authentication, iris authentication, etc. may also be used.

[0052] When fingerprint authentication is used, for example, a fingerprint sensor for detecting the driver's fingerprint may be installed in the vehicle C, the in-vehicle device P, etc. Then, the driver identification unit 210 may, for example, acquire fingerprint information including the driver's fingerprint from the fingerprint sensor and perform fingerprint authentication using the fingerprint information. Fingerprint authentication is a type of biometric authentication using fingerprints, and may be performed by, for example, comparing the feature amount of the driver's fingerprint extracted from the fingerprint information with the feature amount of the driver's fingerprint registered in advance. Note that the fingerprint authentication is not limited to the example given here, and general techniques may be used.

[0053] When using voiceprint authentication, for example, the vehicle C, the in-vehicle device P, etc. may be equipped with a microphone for detecting the driver's voice. The driver identification unit 210 may then acquire, for example, audio information including the driver's voice from the microphone and perform voiceprint authentication using the audio information. Voiceprint authentication is a type of biometric authentication using voice, and may be performed by, for example, comparing features of the driver's voice extracted from the audio information with features of pre-registered drivers' voices. Note that the voiceprint authentication is not limited to the example given here, and general techniques may be used.

[0054] When iris authentication is used, for example, the driver identification unit 210 may acquire an image of the driver from the in-vehicle camera CI and perform iris authentication using the iris (iris image) included in the driver image. Iris authentication is a type of biometric authentication using an iris image, and may be performed by, for example, comparing the iris feature amount of the driver extracted from the iris image with the iris feature amount of pre-registered drivers. Note that the iris authentication is not limited to the example given here, and general techniques may be used.

[0055] (Regarding the first image processing unit 110) As described above, the first image processing unit 110 performs first image processing on a driver image captured of the driver while driving the vehicle C, and generates motion data regarding the driver's motion while driving.

[0056] The first image processing unit 110 may, for example, acquire a driver image from the in-vehicle camera CI and perform first image processing on the acquired driver image to generate operation data. The first image processing unit 110 may, for example, perform the first image processing on the driver image in real time to generate operation data regarding the operation of the driver while driving.

[0057] For example, when generating first individual standard data, the first image processing unit 110 may store a time series of driver images continuously obtained over a predetermined first time length, and then perform first image processing on the stored driver images to generate a history of operation data related to the driver's actions while driving.

[0058] The motion may include, but is not limited to, at least one of eye movement, blinking, eye opening indicating the degree to which the eyes are open, facial direction, hand motion, arm motion, and upper body motion. The motion may include, for example, a combination of hand and arm motion, such as a motion for operating predetermined in-vehicle equipment, such as an air conditioner, audio equipment, or windows.

[0059] The operation may further include vital data such as pulse rate, blood oxygen level, etc. When the vital data is acquired from an image of the driver, the image of the driver is preferably a color image.

[0060] The first image processing may include, for example, processing for detecting a driver's behavior from a driver image. To detect the driver's behavior, for example, a behavior detection model, which is a machine learning model for detecting predetermined behavior from the driver image, may be used. Note that if the behavior data includes vital data, a separate machine learning model for detecting the predetermined vital data from the driver image may be used.

[0061] For example, when a driver image is input, the motion detection model outputs motion detection data indicating the detected driver's motion.

[0062] For example, when the first image processing unit 110 generates motion data in real time, the first image processing unit 110 may acquire driver images in real time, and the driver images may be continuously input to the motion detection model in real time. In this case, the motion detection model may not only detect the motion of the driver while driving using one driver image, but also detect the motion of the driver while driving using driver videos that are time-series driver images.

[0063] Furthermore, for example, when first image processing unit 110 generates motion data using a time series of driver images for a predetermined first time length, the driver images for the predetermined first time length may be input to the motion detection model. In this case, the motion detection model may detect a history of the driver's actions while driving using a driver video that is a group of images of the driver while driving that corresponds to the predetermined first time length.

[0064] The behavior detection model may be constructed by performing learning to detect the behavior of the driver using, for example, training data including a driver image for learning and correct answer data.

[0065] The motion detection model may be included in the first image processing unit 110 or may be provided in an external device (an image analysis device not shown). For example, the first image processing unit 110 may transmit a driver image to the image analysis device and obtain motion detection data generated by the image analysis device using the motion detection model.

[0066] As described above, the motion data is data related to the motion of the driver while driving. In detail, for example, the motion data may include at least one of the following regarding the motion: direction of movement, speed of movement, amount of movement, frequency, and presence or absence of motion. The information included in the motion data may be analyzed by a motion detection model and included in the motion detection data, or may be generated by the first image processing unit 110 using the motion detection data.

[0067] For example, the first image processing unit 110 may perform statistical processing of the movements detected by the movement detection model. That is, the first image processing unit 110 may perform statistical processing using part or all of the movement detection data to generate movement data. This statistical processing may be processing to calculate the frequency, the average amount of movement in a predetermined time, or the like, but is not limited to these. The frequency may be, for example, the number of times a movement is performed within a predetermined time, or the number of times a movement of a predetermined amount or more is performed within a predetermined time.

[0068] In the following, an example will be described in which the action is a gaze movement. The action data may include at least one of the following data related to the gaze movement: the direction of gaze movement, the speed of gaze movement, the amount (or magnitude) of gaze movement, and the frequency of the gaze moving by a predetermined amount or more within a predetermined time. The gaze movement may be detected three-dimensionally, for example, based on a predetermined reference direction. The reference direction is, for example, the optical axis direction (shooting direction) of the in-vehicle camera CI, but is not limited to this.

[0069] (Regarding the first generating unit 120) As described above, the first generating unit 120 generates the first individual standard data based on the history of the operation data. The first individual standard data is data related to standard operations performed by a driver while driving a vehicle.

[0070] For example, in a case where multiple drivers are registered in advance, the first image processing unit 110 may generate first individual standard data for each driver based on the history of the operation data of each of the multiple drivers.

[0071] The first generating unit 120 may store the generated first individual standard data in the determining unit 130 or the like, for example.

[0072] The historical performance data may be generated, for example, using the driver images of the first length of time described above.

[0073] The first individual standard data may be generated, for example, by performing statistical processing using the history of the motion data by the first generating unit 120. In detail, for example, the statistical processing is calculation of an average value, a median value, a maximum value, a minimum value, etc. of a motion (e.g., eye movement) per unit time, but is not limited to these.

[0074] Such first individual standard data is data indicating characteristics of actions performed by a driver while driving a vehicle and is data specific to the driver. For example, by evaluating the current actions performed by the driver while driving using the first individual standard data as a reference, it is possible to appropriately estimate the driver's state, such as whether the driver's state is normal or an abnormal state. For example, the above-mentioned actions may reflect the mental state of the driver who is the subject of the actions. Therefore, the driver's state may also include the driver's mental state. By evaluating the current actions performed by the driver while driving using the first individual standard data as a reference, it is possible to appropriately estimate the driver's mental state, such as whether the driver's mental state is normal or an abnormal state. Examples of abnormal mental states include, but are not limited to, being irritated, agitated, or impatient.

[0075] (Regarding the judgment unit 130) As described above, the judgment unit 130 judges whether the relationship between the operation data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies the output condition.

[0076] For example, the determination unit 130 acquires the first individual standard data generated by the first generation unit 120. The determination unit 130 may acquire the first individual standard data in advance and store it as described above. For example, the determination unit 130 acquires, in real time, the operation data generated in real time by the first generation unit 120.

[0077] Taking the example of a case where the action is gaze movement, the action data may include the direction of gaze movement, the speed of gaze movement, the amount of gaze movement (the amount of gaze movement), and the frequency of gaze movement. The first individual standard data may include, for example, standard values ​​of elements corresponding to elements included in the action data. For example, when average values ​​are used as the standard values, the first individual standard data may include the average values ​​of the direction of gaze movement, the speed of gaze movement, the amount of gaze movement (the amount of gaze movement), and the frequency of gaze movement. The direction may be represented by, for example, a vector, and the average value of the direction may be the average value of the vector representing the direction.

[0078] The relationship between the motion data and the first individual standard data may include, for example, a first degree of difference indicating the degree to which the motion data acquired in real time differs from the first individual standard data.

[0079] The first difference is, for example, a value indicating the degree to which motion data acquired in real time differs from the first individual standard data, using the first individual standard data as a reference. The first difference may include, for example, an element difference for each corresponding element between the motion data and the first individual standard data. For example, the element difference may be a difference, ratio, or the like between the values ​​of corresponding elements between the motion data and the first individual standard data. Furthermore, for example, the first difference may include a value (integrated value) obtained by integrating element differences for multiple corresponding elements between the motion data and the first individual standard data.

[0080] The integration of the element dissimilarity may be performed using, for example, statistical processing, and more specifically, may be performed using, for example, summation, averaging, weighted averaging which is the sum of elements multiplied by predetermined weights, etc. That is, for example, the sum, average, weighted average, etc. of element dissimilarity values ​​for a predetermined number of elements among the plurality of elements included in the motion data and the first individual standard data may be used as the first integrated value.

[0081] The first dissimilarity may be a single value or may include multiple values. When the first dissimilarity includes a single value, the first dissimilarity may include a single dissimilarity related to a predetermined element, or may include a single integrated value obtained by integrating dissimilarity related to multiple predetermined elements. When there are multiple first dissimilarity values, all of the multiple first dissimilarity values ​​may be composed of either element dissimilarity values ​​or integrated values, or may be composed of a combination of element dissimilarity values ​​and integrated values.

[0082] The output conditions are predetermined conditions for outputting output information.

[0083] The output condition may include, for example, a condition related to the first difference. In more detail, for example, the output condition may include a threshold or range related to the first difference. The output condition may include at least one of the following: the first difference is equal to or greater than a threshold, the first difference is less than a threshold, the first difference is within a range, or the first difference is outside a range.

[0084] When the first dissimilarity includes multiple values, multiple output conditions may be defined for each of the multiple values. The thresholds, ranges, etc. used for each output condition may be appropriately determined, and a common threshold, range, etc. may be used, or at least some of the thresholds, ranges, etc. may be different.

[0085] For example, suppose that the corresponding element between the operation data and the first individual standard data includes a gaze fluctuation amplitude, and the first difference degree includes a difference from the average value of the gaze fluctuation amplitude. During normal driving, the driver moves their gaze to some extent. Therefore, the output condition in this case may include that the first difference degree regarding the gaze fluctuation amplitude is outside a predetermined range.

[0086] In this case, for example, the judgment unit 130 judges that the output condition is met if the first difference is outside a predetermined range, and judges that the output condition is not met if the first difference is within the predetermined range.

[0087] In this example, if the output condition is satisfied, the eye movement is either too much or too little compared to normal, so it can be estimated that the driver is in an abnormal state, such as being irritated, agitated, impatient, etc. Therefore, when the determination unit 130 determines that the output condition is satisfied, it may transmit an instruction (output instruction) to the in-vehicle device P to output the determination result or output information.

[0088] If the output condition is not satisfied, the driver's line of sight is moving normally, and it can be assumed that the driver is in a normal state and not irritated, etc. Therefore, for example, when the determination unit 130 determines that the output condition is not satisfied, it may transmit this determination result to the in-vehicle device P, or may terminate information processing without transmitting an output instruction or the like to the in-vehicle device P.

[0089] This allows the in-vehicle device P to output the output information based on the determination result of whether the output condition is satisfied. In detail, for example, the in-vehicle device P may output the output information when it is determined that the output condition is satisfied. The in-vehicle device P may not output the output information when it is determined that the output condition is not satisfied.

[0090] The output information may be, for example, information for alerting a driver to drive safely. The output information may be, for example, notified to a driver from an in-vehicle device P. For example, if the in-vehicle device P has a display unit, the output information may be displayed as a message on the display unit. If the in-vehicle device P has a speaker, the output information may be output as voice, buzzer sound, or the like from the speaker.

[0091] The first image processing unit 110 and the determination unit 130 may repeatedly execute the above-described steps S110b and S130, as illustrated in Fig. 7. Steps S110b and S130 may be repeatedly executed until a predetermined end timing, for example.

[0092] The end timing may be, for example, when the driver leaves the driver's seat of the vehicle C, or when the vehicle C has completed its operation. The driver leaving the driver's seat of the vehicle C may be detected, for example, by using the sensor that detects the person in the driver's seat. The vehicle C may have completed its operation when, for example, the driver gives the vehicle C a predetermined instruction to terminate the operation of the vehicle C.

[0093] According to the present embodiment, the information processing device 200 includes the driver identification unit 210 that identifies the driver who is driving. The first generation unit 120 generates first individual standard data for each driver based on the history of the operation data of each of the multiple drivers.

[0094] As a result, even if one of a plurality of drivers is driving, it is possible to appropriately determine whether or not to output the output information based on the driver image captured of the driver driving the vehicle, thereby making it possible to easily support safe driving.

[0095] According to this embodiment, the movements include at least one of eye movement, blink frequency, eye opening indicating the degree to which the eyes are open, face direction, hand movement, arm movement, and upper body movement.

[0096] These actions may generally be performed depending on the state of the driver. Therefore, by generating action data relating to such actions from a driver image, it is possible to appropriately determine whether or not to output output information depending on the driver while driving. Therefore, it is possible to easily support safe driving.

[0097] According to this embodiment, the operation data further includes vital data of the driver while driving.

[0098] Generally, vital data may change depending on the state of the driver. Therefore, by generating operation data including vital data from a driver image, it is possible to appropriately determine whether to output information depending on the driver while driving. Therefore, it is possible to easily support safe driving.

[0099] [Embodiment 3] In this embodiment, an example will be described in which the driver's state is estimated by further using an outside image captured outside the vehicle C. Note that in this embodiment, for the sake of simplicity, descriptions that overlap with other embodiments will be omitted as appropriate.

[0100] 8, the information processing system S3 includes an in-vehicle camera CI, an in-vehicle device P, and an exterior camera CO mounted on a vehicle C, and an information processing device 300. The in-vehicle camera CI and the in-vehicle device P may be similar to those described in the other embodiments.

[0101] (Regarding the exterior camera CO) The exterior camera CO is an example of an imaging device that captures images outside the vehicle. The vehicle C may be provided with a plurality of exterior cameras CO.

[0102] Here, outside the vehicle means the outside of the vehicle C. The outside camera CO is installed on the vehicle C so as to capture a predetermined range around the vehicle C, such as the front, rear, and sides of the vehicle C, and captures images of the outside of the vehicle C while the driver is driving.

[0103] The exterior camera CO, for example, transmits exterior images of the outside of the vehicle to the information processing device 300. For example, the exterior camera CO may continuously capture images of the outside of the vehicle and transmit the exterior images obtained by the capture to the information processing device 300 in real time.

[0104] The exterior camera CO may be any of a variety of imaging devices, for example, a camera that generates monochrome images or a camera that generates color images.

[0105] (Regarding the information processing device 300) As shown in FIG. 9 , the information processing device 300 includes a second image processing unit 310, the driver identification unit 210 and the first image processing unit 110 described above, a first generation unit 320 that replaces the first generation unit 120, and a judgment unit 330 that replaces the judgment unit 130.

[0106] The second image processing unit 310 performs second image processing on an outside image captured outside the vehicle C to detect an object outside the vehicle.

[0107] The first generation unit 320 generates first individual standard data regarding standard actions that a driver performs while driving a vehicle in response to objects outside the vehicle, based on the history of operation data and the history of detected objects outside the vehicle.

[0108] The judgment unit 330 judges whether the relationship between the operation data generated in real time for the driver while driving and the objects outside the vehicle detected in real time, and the first individual standard data for the driver while driving satisfies the output conditions.

[0109] The output conditions include conditions regarding the driver's actions while driving in response to objects outside the vehicle.

[0110] (Example of Processing Operation of Information Processing Device 300) The information processing device 300 executes, for example, information processing as shown in Fig. 10. This information processing is another example of processing for generating first individual standard data.

[0111] The first image processing unit 110 executes the above-mentioned step S110a.

[0112] The second image processing unit 310 performs second image processing on an exterior image captured outside the vehicle C to detect an object outside the vehicle (step S310a). In step S310a, for example, the second image processing unit 310 may perform second image processing on a history of exterior images captured during a predetermined first time length to detect a history of objects outside the vehicle.

[0113] The first generation unit 320 generates first individual standard data regarding standard actions that a driver performs while driving a vehicle in response to objects outside the vehicle, based on the history of operation data and the history of detected objects outside the vehicle (step S320).

[0114] The information processing device 300 executes, for example, information processing as shown in Fig. 11. This information processing is another example of processing for determining whether or not an output condition is satisfied.

[0115] The driver identification unit 210 executes the above-mentioned step S210.

[0116] The first image processing unit 110 executes the above-mentioned step S110b.

[0117] The second image processing unit 310 performs second image processing on the exterior image captured outside the vehicle C to detect an object outside the vehicle (step S310b). In step S310b, for example, the second image processing unit 310 may perform second image processing on the exterior image captured outside the vehicle in real time to detect an object outside the vehicle.

[0118] The judgment unit 330 judges whether the relationship between the operation data generated in real time for the driver while driving and the object outside the vehicle detected in real time, and the first individual standard data for the driver while driving satisfies the output conditions for outputting predetermined output information (step S330).

[0119] Hereinafter, a detailed example of this embodiment will be described using an example in which the movement is a line of sight, but a movement other than a line of sight may also be used.

[0120] (Second Image Processing Unit 310) As described above, the second image processing unit 310 performs second image processing on an outside image captured outside the vehicle C, and detects an object outside the vehicle.

[0121] The second image processing unit 310 may, for example, acquire an exterior image from the exterior camera CO and perform second image processing on the acquired exterior image to detect an object outside the vehicle. The second image processing unit 310 may, for example, perform second image processing on the exterior image in real time to detect an object outside the vehicle.

[0122] For example, when generating first individual standard data, the second image processing unit 310 may store a time series of exterior vehicle images obtained continuously over a predetermined first time length, and perform second image processing on the stored exterior vehicle images to generate a history of objects outside the vehicle.

[0123] The second image processing may include, for example, processing for detecting an object outside the vehicle from the vehicle exterior image. To detect the object outside the vehicle, for example, an object detection model, which is a machine learning model for detecting a predetermined object outside the vehicle from the vehicle exterior image, may be used.

[0124] The predetermined object outside the vehicle may include, but is not limited to, at least one of a traffic light, a stop line, a dividing line, a traffic sign, a signboard, a fallen object on the road, a moving object, etc. The dividing line may include, for example, a center line indicating the boundary between the oncoming lane and the own lane, a lane boundary line indicating the boundary between lanes in the same direction, etc. The moving object may include, for example, a person such as a pedestrian, or another vehicle such as another car, a motorcycle, or a bicycle.

[0125] The object detection model receives an outside-of-vehicle image as input, and outputs object detection data indicating an object included in the outside-of-vehicle image. The object detection data may include at least one of the object position, the object type, and the like.

[0126] For example, when the second image processing unit 310 generates object detection data in real time, the second image processing unit 310 may acquire vehicle exterior images in real time, and the vehicle exterior images may be continuously input to the object detection model in real time. In this case, the object detection model may not only detect an object outside the vehicle using a single vehicle exterior image, but also detect an object outside the vehicle using vehicle exterior video, which is a time series of vehicle exterior images.

[0127] For example, when the second image processing unit 310 generates object detection data using a time series of vehicle exterior images over a predetermined first time length, the vehicle exterior images over the predetermined first time length may be input to the object detection model. In this case, the object detection model may detect the history of objects outside the vehicle using vehicle exterior video, which is a group of vehicle exterior images corresponding to the predetermined first time length.

[0128] The object detection model may be constructed by performing learning to detect the driver's actions using training data including, for example, driver images for learning and correct answer data.

[0129] The motion detection model may be included in the second image processing unit 310 or may be provided in an external device (an image analysis device not shown). For example, the second image processing unit 310 may transmit an image of the outside of the vehicle to the image analysis device, and acquire object detection data generated by the image analysis device using the object detection model. Note that the image analysis device that includes the motion detection model and the object detection model may be the same or different.

[0130] (Regarding the first generating unit 320) As described above, the first generating unit 320 generates the first individual standard data based on, for example, the history of the operation data and the history of the detected object outside the vehicle. The first individual standard data in this embodiment may include data on standard operations that the driver performs while driving the vehicle in response to the object outside the vehicle.

[0131] The first generating section 320 may store the generated first individual standard data in the determining section 330 or the like, for example.

[0132] The operation data history may be generated using, for example, a driver image of a first length of time as described above. The vehicle exterior object history may be generated using, for example, the vehicle exterior image of the first length of time as described above. The first image processing for generating the driver operation data history and the second image processing for detecting the vehicle exterior object history may use driver images and vehicle exterior images captured at substantially the same time.

[0133] The first individual standard data may be generated, for example, by performing statistical processing using the history of the motion data and the history of the detected object outside the vehicle by the first generating unit 320. In detail, for example, the statistical processing is calculation of an average value, a median value, a maximum value, a minimum value, etc. of the motion (e.g., eye movement) per unit time, but is not limited to these.

[0134] For example, a driver may move his / her eyes while driving the vehicle in response to an object outside the vehicle. The first individual standard data may include, for example, data on standard actions taken toward an object outside the vehicle while driving the vehicle, for each type of object outside the vehicle. In detail, for example, if the object outside the vehicle is a person within a predetermined range from the vehicle C, the first individual standard data may include, for example, the average number of times the driver turns his / her eyes to each person within the predetermined range from the vehicle C.

[0135] The first individual standard data is data indicating the characteristics of the actions performed by the currently driving driver while driving the vehicle, and is data specific to the currently driving driver. Therefore, as described above, for example, by using the first individual standard data as a reference to evaluate the current actions performed by the currently driving driver, the driver's state can be appropriately estimated.

[0136] (Regarding the judgment unit 330) As described above, the judgment unit 330 judges whether the relationship between the operation data generated in real time for the driver while driving and the objects outside the vehicle detected in real time, and the first individual standard data for the driver while driving, satisfies the output conditions.

[0137] For example, the determination unit 330 acquires the first individual standard data generated by the first generation unit 320. The determination unit 330 may acquire the first individual standard data in advance and store it as described above. For example, the determination unit 330 acquires, in real time, the operation data generated in real time by the first generation unit 120 and objects outside the vehicle detected in real time by the second image processing unit 310.

[0138] The determination unit 330 calculates a value related to the driver's behavior toward an object outside the vehicle while driving, for example, by using the behavior data generated in real time and the object outside the vehicle detected in real time. The value related to the behavior toward the object outside the vehicle may be, for example, a value of an element corresponding to an element included in the first individual standard data. In detail, for example, the value related to the behavior toward the object outside the vehicle may be, for example, the number of times the driver looks at a person within a predetermined range from the vehicle C.

[0139] The relationship between the first individual standard data and the operation data and the object outside the vehicle may include, for example, the above-mentioned first difference degree. In detail, for example, the first difference degree may be a difference between the number of times the driver looks at a person within a predetermined range from the vehicle C while driving and the average number of times included in the first individual standard data (difference in the number of times of looking).

[0140] The output conditions may include, for example, conditions regarding the driver's behavior while driving in response to objects outside the vehicle.

[0141] In detail, for example, the elements corresponding between the operation data, the object outside the vehicle, and the first individual standard data include the number of times the driver looks at a person within a predetermined range from the vehicle C, and the first difference degree includes the difference in the number of times the driver looks at a person within a predetermined range from the vehicle C. In normal driving, the driver looks at a person within a predetermined range from the vehicle C a certain number of times to check on them. Therefore, the output condition in this case may be that the first difference degree is equal to or less than a predetermined threshold value.

[0142] In this case, for example, the determination unit 330 determines that the output condition is met when the first difference is equal to or less than a predetermined threshold, and determines that the output condition is not met when the first difference is greater than the predetermined threshold.

[0143] If the output condition is satisfied, it can be estimated that the driver is in an abnormal state. Therefore, for example, when the determination unit 330 determines that the output condition is satisfied, the determination unit 330 may transmit an instruction (output instruction) to the in-vehicle device P to output the determination result or output information.

[0144] If the output condition is not satisfied, the gaze is moving normally, so it can be assumed that the state is normal and there is no irritation, etc. Therefore, for example, when the determination unit 330 determines that the output condition is not satisfied, the determination unit 330 may transmit this determination result to the in-vehicle device P, or may terminate information processing without transmitting an output instruction, etc. to the in-vehicle device P.

[0145] This allows the in-vehicle device P to output the output information based on the determination result of whether or not the output condition is satisfied.

[0146] The first image processing unit 110, the second image processing unit 310, and the determination unit 330 may repeatedly execute the above-described steps S110b, S310b, and S330, as illustrated in Fig. 11. Steps S110b, S310b, and S130 may be repeatedly executed, for example, until the above-described predetermined end timing.

[0147] (Operations and Effects) As described above, according to this embodiment, the information processing device 300 includes the second image processing unit 310 that performs second image processing on an exterior image captured outside the vehicle to detect an object outside the vehicle.

[0148] Furthermore, the first generating unit 320 generates first individual standard data related to standard actions performed by the driver while driving the vehicle in response to objects outside the vehicle, based on the history of the action data and the history of detected objects outside the vehicle. The output condition includes a condition related to the driver's actions while driving in response to objects outside the vehicle. The determining unit 330 determines whether the relationship between the action data generated in real time for the driver while driving and the objects outside the vehicle detected in real time, and the first individual standard data for the driver while driving satisfies the output condition.

[0149] This allows the system to appropriately determine whether to output the output information based on the driver's behavior toward an object outside the vehicle. Generally, the driver may perform a behavior toward an object outside the vehicle. Therefore, it is possible to easily and appropriately support safe driving.

[0150] The action includes the movement of the eyes.

[0151] This allows the system to appropriately determine whether to output output information based on the driver's line of sight while driving relative to an object outside the vehicle. Generally, drivers often move their line of sight relative to an object outside the vehicle. Therefore, it is possible to easily and appropriately support safe driving.

[0152] [Embodiment 4] In this embodiment, an example in which the driving environment of the vehicle is further used to estimate the driver's state will be described. Note that, in this embodiment, for the sake of simplicity, descriptions that overlap with other embodiments will be omitted as appropriate.

[0153] The information processing device 400 is an information processing device that is provided in the information processing systems S1, S2, and S3, for example, in place of the information processing devices 100, 200, and 300. The information processing device 400 includes a driving environment acquisition unit 410, as shown in Fig. 12, for example. The information processing device 400 also includes the driver identification unit 210 and the first image processing unit 110 described above, a first generation unit 420 that replaces the first generation units 120 and 320, and a determination unit 430 that replaces the determination units 130 and 330.

[0154] When the information processing device 400 is applied to the information processing system S2, the information processing device 400 may further include a second image processing unit 310. Furthermore, the first generating unit 420 and the determining unit 430 may further have the functions of the first generating unit 320 and the determining unit 330.

[0155] The driving environment acquisition unit 410 acquires driving environment data indicating the driving environment of the vehicle C.

[0156] The first generating unit 420 generates first individual standard data relating to standard actions that a driver performs while driving a vehicle in accordance with the driving environment, based on the history of the action data and the history of the driving environment data.

[0157] The judgment unit 430 judges whether the relationship between the operation data generated in real time for the driver while driving and the driving environment data acquired in real time and the first individual standard data for the driver while driving satisfies the output condition.

[0158] When applied to the information processing system S2, the first generation unit 420 may generate first individual standard data regarding standard actions performed by a driver while driving a vehicle in accordance with objects outside the vehicle and the driving environment, based on the history of action data, the history of detected objects outside the vehicle, and the history of driving environment data.When applied to the information processing system S2, the determination unit 430 determines whether the relationship between the action data, the objects outside the vehicle, and the driving environment data and the first individual standard data for the driver while driving satisfies an output condition.The action data, the objects outside the vehicle, and the driving environment data used here may be action data generated in real time for the driver while driving, objects outside the vehicle detected in real time, and driving environment data acquired in real time, respectively.

[0159] (Example of Processing Operation of Information Processing Device 400) The information processing device 400 executes, for example, information processing as shown in Fig. 13. This information processing is another example of processing for generating first individual standard data.

[0160] The first image processing unit 110 executes the above-mentioned step S110a.

[0161] The driving environment acquisition unit 410 acquires driving environment data indicating the driving environment of the vehicle C (step S410a). In step S410a, for example, the driving environment acquisition unit 410 may acquire a history of driving environment data for a period corresponding to the time when the driver image used in step S110a was captured.

[0162] The first generation unit 420 generates first individual standard data regarding standard actions that a driver performs while driving a vehicle in accordance with the driving environment based on the history of operation data and the history of driving environment data (step S420).

[0163] The information processing device 400 executes, for example, information processing as shown in Fig. 14. This information processing is another example of processing for determining whether or not an output condition is satisfied.

[0164] The driver identification unit 210 executes the above-mentioned step S210.

[0165] The first image processing unit 110 executes the above-mentioned step S110b.

[0166] The driving environment acquisition unit 410 acquires driving environment data indicating the driving environment of the vehicle C (step S410b). In step S410b, for example, the driving environment acquisition unit 410 may acquire the driving environment data indicating the driving environment of the vehicle C in real time.

[0167] The judgment unit 430 judges whether the relationship between the operation data generated in real time for the driver while driving and the driving environment data acquired in real time and the first individual standard data for the driver while driving satisfies the output condition (step S430).

[0168] (Regarding the driving environment acquisition unit 410) The driving environment acquisition unit 410 acquires, for example, driving environment data indicating the driving environment in which the vehicle C is driving. For example, the driving environment acquisition unit 410 acquires the current position of the vehicle C from the in-vehicle device P and acquires driving environment data for the current position using, for example, map data. The map data may be acquired from an external device (not shown). In this case, the in-vehicle device P may have a function for acquiring the current position using, for example, a GPS (Global Positioning System). Note that the method for acquiring the current position is not limited to the method exemplified here, and the current position may also be acquired using, for example, an outside-vehicle image.

[0169] The driving environment may be, for example, a type of situation around the vehicle C, and in detail, may include at least one of, for example, the type of road on which the vehicle C is traveling, the type of road situation, etc. The type of road may be, for example, an ordinary road, an expressway, etc., but is not limited to these. The type of road situation may be, for example, a crowded situation where there are a predetermined number of people or more around the vehicle C, a general situation where there are less than a predetermined number of people around the vehicle C, a crowded situation where there are a predetermined number of other vehicles around the vehicle C, or a situation requiring particular attention to the surroundings, such as frequent occurrences of people running out into the road, but is not limited to these.

[0170] (Regarding the first generating unit 420) As described above, the first generating unit 420 generates the first individual standard data based on the history of the operation data and the history of the driving environment data. The first individual standard data in this embodiment may include data on standard operations that the driver performs while driving the vehicle in accordance with the driving environment.

[0171] The first generating section 420 may store the generated first individual standard data in the determining section 430 or the like, for example.

[0172] The first individual standard data may be generated, for example, by performing statistical processing using the history of the operation data and the history of the acquired driving environment data by the first generating unit 420. In detail, for example, the statistical processing is calculation of an average value, a median value, a maximum value, a minimum value, etc. of the operation (e.g., eye movement) per unit time, but is not limited to these.

[0173] For example, a driver may perform different actions while driving depending on the type of situation around the vehicle C. The first individual standard data may include data on standard actions performed while driving the vehicle for each type of situation around the vehicle C. In more detail, for example, the types of situations around the vehicle C may include ordinary roads and expressways, and the actions may be the amplitude of gaze fluctuation. In this case, the first individual standard data may include an average value of the amplitude of gaze fluctuation per unit time on ordinary roads, an average value of the amplitude of gaze fluctuation per unit time on expressways, etc.

[0174] The first individual standard data is data indicating the characteristics of the actions performed by the currently driving driver while driving the vehicle, and is data specific to the currently driving driver. Therefore, as described above, for example, by using the first individual standard data as a reference to evaluate the current actions performed by the currently driving driver, the driver's state can be appropriately estimated.

[0175] (Regarding the judgment unit 430) As described above, the judgment unit 430 judges whether the relationship between the operation data generated in real time for the driver while driving and the driving environment data acquired in real time for the driver while driving, and the first individual standard data for the driver while driving, satisfies the output condition.

[0176] For example, the determination unit 430 acquires the first individual standard data generated by the first generation unit 420. The determination unit 430 may acquire the first individual standard data in advance and store it as described above. For example, the determination unit 430 acquires, in real time, the operation data generated in real time by the first generation unit 120 and the driving environment data acquired in real time by the driving environment acquisition unit 410.

[0177] The determination unit 430 calculates a value related to the behavior of the driver according to the driving environment, for example, by using the behavior data generated in real time and the driving environment data acquired in real time. The value related to the behavior according to the driving environment may be, for example, a value of an element corresponding to an element included in the first individual standard data. In detail, for example, when the vehicle C is traveling on a highway and the behavior is the amplitude of gaze fluctuation, the value related to the behavior according to the driving environment may be a value related to the behavior on the highway, which may be an average value of the amplitude of gaze fluctuation per unit time.

[0178] The relationship between the operation data and the first individual standard data may include, for example, the first difference degree described above. In detail, for example, the first difference degree may be the difference between the average value of the gaze fluctuation amplitude on the expressway calculated in real time and the average value of the gaze fluctuation amplitude on the expressway included in the first individual standard data.

[0179] The output conditions may include, for example, conditions regarding the driver's behavior while driving depending on the driving environment.

[0180] In more detail, for example, the elements corresponding to the operation data and the driving environment on the one hand and the first individual standard data on the other hand include an average value of the gaze fluctuation amplitude on a highway, and the first difference degree includes the difference between the average values ​​of the gaze fluctuation amplitude on the highway. Normal driving on a highway, unlike on ordinary roads, involves driving on roads with no pedestrians and with a relatively good view at higher speeds than on ordinary roads. Therefore, if the gaze fluctuation amplitude is large, there is a possibility that the driver is not concentrating on driving the vehicle C. Therefore, the output condition in this case may be that the first difference degree is equal to or greater than a predetermined threshold value.

[0181] In this case, for example, the judgment unit 430 judges that the output condition is met if the first difference is greater than or equal to a predetermined threshold, and judges that the output condition is not met if the first difference is less than the predetermined threshold.

[0182] If the output condition is satisfied, it can be estimated that the driver is not concentrating on driving and is in an abnormal state. Therefore, for example, when the determination unit 430 determines that the output condition is satisfied, the determination unit 430 may transmit an instruction (output instruction) to the in-vehicle device P to output the determination result or output information.

[0183] If the output condition is not satisfied, it can be assumed that the driver is concentrating on driving and is in a normal state. Therefore, for example, when the determination unit 430 determines that the output condition is not satisfied, the determination unit 430 may transmit this determination result to the in-vehicle device P, or may terminate information processing without transmitting an output instruction or the like to the in-vehicle device P.

[0184] This allows the in-vehicle device P to output the output information based on the determination result of whether the output condition is satisfied. In detail, for example, the in-vehicle device P may output the output information when it is determined that the output condition is satisfied.

[0185] The first image processing unit 110, the driving environment acquisition unit 410, and the determination unit 430 may repeatedly execute the above-described steps S110b, S410b, and S430, as illustrated in Fig. 14. Steps S110b, S410b, and S430 may be repeatedly executed, for example, until the above-described predetermined end timing.

[0186] (Actions and Effects) As described above, according to this embodiment, the information processing device 400 includes a driving environment acquisition unit 410 that acquires driving environment data indicating the driving environment of the vehicle C. The first generation unit 420 generates first individual standard data related to standard actions performed by the driver while driving the vehicle in accordance with the driving environment, based on the history of action data and the history of driving environment data. The output condition includes a condition related to the driver's actions while driving in accordance with the driving environment. The determination unit 430 determines whether the relationship between the action data generated in real time for the driver while driving and the driving environment data acquired in real time, and the first individual standard data for the driver while driving, satisfies the output condition.

[0187] This allows the system to appropriately determine whether to output the output information based on the driver's behavior during driving in accordance with the vehicle's driving environment. Generally, the driver may perform different actions depending on the vehicle's driving environment. Therefore, it is possible to easily and appropriately support safe driving.

[0188] (Example of physical configuration of information processing devices 100, 200, 300, 400, 500) The information processing devices 100, 200, 300, 400, 500 physically include a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070, as shown in FIG. 15, for example.

[0189] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0190] The processor 1020 is implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0191] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0192] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read-only memory (ROM), or the like. The storage device 1040 stores program modules for realizing the functions of the device that includes the storage device 1040. The processor 1020 loads each of these program modules into the memory 1030 and executes them to realize the function corresponding to that program module.

[0193] The network interface 1050 is an interface for connecting a device equipped with it to a communication network.

[0194] The input interface 1060 is an interface for the user to input information, and is configured from, for example, a touch panel, a keyboard, a mouse, and the like.

[0195] The output interface 1070 is an interface for presenting information to the user, and is configured, for example, by a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like.

[0196] In this way, the functions of the information processing devices 100, 200, 300, 400, and 500 can be realized by the physical components cooperating to execute a software program. Therefore, the present invention may be realized as a software program or as a non-transitory storage medium on which the program is recorded. Note that the information processing device may be physically composed of multiple devices (e.g., computers, etc.).

[0197] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0198] In addition, although the flowcharts used in the above description show a sequence of steps (processes), the order of steps executed in each embodiment is not limited to the sequence shown in the flowcharts. In each embodiment, the order of steps shown in the diagrams can be changed as long as it does not cause any problems in terms of the content.

[0199] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. 1. An information processing device comprising: first image processing means that performs first image processing on a driver image captured of a driver driving a vehicle to generate behavior data related to the behavior of the driver while driving; first generation means that generates first individual standard data related to standard behavior performed by the driver while driving the vehicle based on a history of the behavior data; and determination means that determines whether a relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies an output condition for outputting predetermined output information. 2. The information processing device described in 1., further comprising driver identification means that identifies the driver while driving, wherein the first generation means generates the first individual standard data for each driver based on a history of the behavior data of each of a plurality of drivers. 3. The information processing device according to 1. or 2., further comprising second image processing means for performing second image processing on an outside image taken outside the vehicle to detect an object outside the vehicle, wherein the first generation means generates the first individual standard data related to a standard action taken by the driver while driving the vehicle in accordance with the object outside the vehicle based on a history of the action data and a history of the detected object outside the vehicle, wherein the output condition includes a condition related to the action of the driver while driving in accordance with the object outside the vehicle, and the determination means determines whether or not the relationship between the action data generated in real time for the driver while driving and the object outside the vehicle detected in real time, and the first individual standard data for the driver while driving satisfies the output condition. 4. The information processing device according to 3., wherein the action includes a movement of the driver's line of sight.5. The information processing device according to any one of 1 to 4, further comprising driving environment acquisition means for acquiring driving environment data indicating a driving environment of the vehicle, wherein the first generation means generates the first individual standard data related to standard actions performed by the driver while driving the vehicle in accordance with the driving environment based on a history of the action data and a history of the driving environment data, wherein the output condition includes a condition related to the driver's action while driving in accordance with the driving environment, and the determination means determines whether or not a relationship between the action data generated in real time for the driver while driving and the driving environment data acquired in real time for the driver while driving, and the first individual standard data for the driver while driving satisfies the output condition. 6. The information processing device according to any one of 1 to 5, wherein the actions include at least one of eye movement, blinking frequency, eye opening indicating the degree to which the eyes are open, face direction, hand movement, arm movement, and upper body movement. 7. The information processing device according to any one of 1 to 6, wherein the action data further includes vital sign data of the driver while driving. 8. An information processing system comprising: an information processing device; and an in-vehicle terminal mounted on a vehicle, wherein the information processing device comprises: a first image processing means that performs first image processing on a driver image captured of the driver while driving the vehicle to generate behavior data related to the behavior of the driver while driving; a first generation means that generates first individual standard data related to standard behavior performed by the driver while driving the vehicle based on a history of the behavior data; and a judgment means that judges whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies an output condition for outputting predetermined output information, and the in-vehicle terminal outputs the output information based on the judgment result of whether the output condition is satisfied.9. An information processing method in which one or more computers perform first image processing on a driver image captured of a driver driving a vehicle to generate behavior data related to the behavior of the driver while driving, generate first individual standard data related to standard behavior performed by the driver while driving the vehicle based on a history of the behavior data, and determine whether a relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies an output condition for outputting predetermined output information. 10. The information processing method described in 9. further includes, in identifying the driver while driving and generating the first individual standard data, generating the first individual standard data for each driver based on a history of the behavior data of each of a plurality of drivers. 11. The information processing method according to 9. or 10. further includes performing second image processing on an exterior image taken outside the vehicle to detect an object outside the vehicle, generating the first individual standard data includes generating the first individual standard data related to a standard action taken by the driver while driving the vehicle in response to an object outside the vehicle based on a history of the action data and a history of the detected object outside the vehicle, the output condition includes a condition related to an action of the driver while driving in response to an object outside the vehicle, and determining whether the output condition is satisfied includes determining whether a relationship between the action data generated in real time for the driver while driving and the object outside the vehicle detected in real time, and the first individual standard data for the driver while driving satisfies the output condition. 12. The information processing method according to 11., wherein the action includes a movement of gaze.13. The information processing method according to any one of 9 to 12, further comprising: acquiring driving environment data indicating a driving environment of the vehicle; generating the first individual standard data includes generating the first individual standard data related to standard actions performed by the driver while driving the vehicle in accordance with the driving environment based on a history of the action data and a history of the driving environment data; the output condition includes a condition related to the driver's action while driving in accordance with the driving environment; and determining whether the output condition is satisfied includes determining whether a relationship between the action data generated in real time for the driver while driving and the driving environment data acquired in real time, and the first individual standard data for the driver while driving satisfies the output condition. 14. The information processing method according to any one of 9 to 13, wherein the action includes at least one of eye movement, blinking frequency, eye opening indicating the degree to which the eyes are open, face direction, hand movement, arm movement, and upper body movement. 15. The information processing method according to any one of 9 to 14, wherein the action data further includes vital sign data of the driver while driving. 16. 17. A program for causing one or more computers to execute the following steps: performing first image processing on a driver image captured of a driver while driving a vehicle to generate behavior data related to the behavior of the driver while driving, generating first individual standard data related to standard behavior performed by the driver while driving the vehicle based on a history of the behavior data, and determining whether a relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies an output condition for outputting predetermined output information. 17. The program described in 16. further causes the computer to execute identifying the driver while driving, wherein generating the first individual standard data generates the first individual standard data for each driver based on a history of the behavior data of each of a plurality of drivers.18. The program according to 16. or 17., further comprising: performing second image processing on an exterior image captured outside the vehicle to detect an object outside the vehicle; generating the first individual standard data includes generating the first individual standard data related to a standard action taken by the driver while driving the vehicle in response to an object outside the vehicle based on a history of the action data and a history of the detected object outside the vehicle; the output condition includes a condition related to an action of the driver while driving in response to an object outside the vehicle; and determining whether the output condition is satisfied includes determining whether a relationship between the action data generated in real time for the driver while driving and the object outside the vehicle detected in real time, and the first individual standard data for the driver while driving satisfies the output condition. 19. The program according to 18., wherein the action includes a movement of gaze. 20. The program according to any one of 16 to 19, further comprising: acquiring driving environment data indicating a driving environment of the vehicle; generating the first individual standard data includes generating the first individual standard data related to standard actions performed by the driver while driving the vehicle in accordance with the driving environment based on a history of the action data and a history of the driving environment data; the output condition includes a condition related to the driver's action while driving in accordance with the driving environment; and determining whether the output condition is satisfied includes determining whether a relationship between the action data generated in real time for the driver while driving and the driving environment data acquired in real time, and the first individual standard data for the driver while driving satisfies the output condition. 21. The program according to any one of 16 to 20, wherein the action includes at least one of eye movement, blink frequency, eye opening indicating a degree to which the eyes are open, face direction, hand movement, arm movement, and upper body movement. 22. The program according to any one of 16 to 21, wherein the action data further includes vital sign data of the driver while driving. 23. A recording medium on which the program according to any one of 16 to 22 is recorded.

[0200] This application claims priority based on Japanese Patent Application No. 2024-069525, filed April 23, 2024, the disclosure of which is incorporated herein by reference in its entirety.

[0201] S1, S2, S3 Information processing system C Vehicle P In-vehicle device CI In-vehicle camera CO Out-vehicle camera 100, 200, 300, 400, 500 Information processing device 110 First image processing unit 120, 320, 420 First generation unit 130, 330, 430 Determination unit 210 Driver identification unit 310 Second image processing unit 410 Traveling environment acquisition unit

Claims

1. An information processing device comprising: a first image processing means that performs first image processing on a driver image captured of a driver while driving a vehicle to generate behavior data related to the behavior of the driver while driving; a first generation means that generates first individual standard data related to standard behavior performed by the driver while driving a vehicle based on a history of the behavior data; and a judgment means that judges whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies output conditions for outputting predetermined output information.

2. The information processing device according to claim 1, further comprising a driver identification means for identifying the driver currently driving, wherein the first generation means generates the first individual standard data for each of the plurality of drivers based on a history of the operation data of each of the plurality of drivers.

3. An information processing device as described in claim 1 or 2, further comprising a second image processing means for performing second image processing on an exterior image taken outside the vehicle to detect objects outside the vehicle, wherein the first generation means generates the first individual standard data regarding standard actions taken by the driver while driving the vehicle in response to objects outside the vehicle based on the history of the action data and the history of the detected objects outside the vehicle, wherein the output conditions include conditions regarding the actions of the driver while driving in response to objects outside the vehicle, and wherein the judgment means judges whether the relationship between the action data generated in real time for the driver while driving and the objects outside the vehicle detected in real time, and the first individual standard data for the driver while driving satisfies the output conditions.

4. The information processing device according to claim 3, wherein the action includes eye movement.

5. An information processing device as described in any one of claims 1 to 4, further comprising a driving environment acquisition means for acquiring driving environment data indicating the driving environment of the vehicle, wherein the first generation means generates the first individual standard data regarding standard actions performed by the driver while driving the vehicle in accordance with the driving environment based on the history of the action data and the history of the driving environment data, wherein the output conditions include conditions regarding the actions of the driver while driving in accordance with the driving environment, and the judgment means judges whether the relationship between the action data generated in real time for the driver while driving and the driving environment data acquired in real time, and the first individual standard data for the driver while driving satisfies the output conditions.

6. An information processing device according to any one of claims 1 to 5, wherein the movements include at least one of eye movement, blinking frequency, eye opening indicating the degree to which the eyes are open, facial direction, hand movement, arm movement, and upper body movement.

7. The information processing device according to any one of claims 1 to 6, wherein the operation data further includes vital data of the driver while driving.

8. An information processing system comprising an information processing device and an in-vehicle terminal mounted on a vehicle, wherein the information processing device comprises: a first image processing means that performs first image processing on a driver image captured of the driver while driving the vehicle to generate behavior data related to the behavior of the driver while driving; a first generation means that generates first individual standard data related to standard behavior performed by the driver while driving the vehicle based on a history of the behavior data; and a judgment means that judges whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies an output condition for outputting predetermined output information, and the in-vehicle terminal outputs the output information based on the judgment result of whether the output condition is satisfied.

9. An information processing method in which one or more computers perform first image processing on a driver image taken of a driver while driving a vehicle to generate behavior data regarding the behavior of the driver while driving, generate first individual standard data regarding standard behavior performed by the driver while driving the vehicle based on a history of the behavior data, and determine whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies output conditions for outputting predetermined output information.

10. A recording medium having recorded thereon a program for causing one or more computers to perform the following: performing first image processing on a driver image taken of a driver while driving a vehicle to generate behavior data regarding the behavior of the driver while driving; generating first individual standard data regarding standard behavior performed by the driver while driving a vehicle based on the history of the behavior data; and determining whether the relationship between the behavior data generated in real time for the driver while driving and the first individual standard data for the driver while driving satisfies output conditions for outputting predetermined output information.

Images