Information processing method, program, and information processing apparatus

Abstract

To provide an information processing method or the like capable of efficiently generating reference data when imaging a driver of a vehicle by an imaging device and performing monitoring processing to an operator.SOLUTION: An information processing method causes a computer to execute processing of acquiring an image including an operator who operates a vehicle by a camera mounted on the vehicle, determining whether or not the vehicle is in a predetermined traveling state, and generating reference data when performing monitoring processing on the operator on the basis of the acquired image when it is determined that the vehicle is in the predetermined traveling state.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing device. [Background technology]

[0002] A safe driving judgment device is known that captures an image of a vehicle driver using an imaging device, obtains an angle value indicating the angle of the driver's face when based on the vehicle's direction of travel, and determines whether the driver is not paying attention to the road ahead based on the obtained angle value (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-144421 Summary of the Invention [Problem to be solved by the invention]

[0004] In one aspect, an object is to provide an information processing method and the like that can efficiently generate reference data when performing monitoring processing on an operator. [Means for solving the problem]

[0005] An information processing method according to one aspect of the present disclosure acquires an image including an operator operating the vehicle using a camera mounted on the vehicle, determines whether the vehicle is in a predetermined driving state, and if it is determined that the vehicle is in the predetermined driving state, causes a computer to execute a process of generating reference data for performing monitoring processing on the operator based on the acquired image. [Effects of the Invention]

[0006] According to one aspect of the present disclosure, it is possible to provide an information processing method or the like that efficiently generates reference data when performing a monitoring process on an operator. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram illustrating the configuration of a camera unit including an information processing device (camera ECU) according to a first embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a physical configuration of an information processing apparatus. [Figure 3] FIG. 10 is an explanatory diagram relating to a generation process of a face detection model. [Figure 4] FIG. 10 is an explanatory diagram relating to a generation process of a facial landmark model. [Figure 5] FIG. 10 is an explanatory diagram regarding a generation process of a posture detection model. [Figure 6] 10 is a flowchart illustrating a process (calibration process) of a control unit of the information processing device. [Figure 7] FIG. 10 is an explanatory diagram showing the detection result of the position (bounding box) of a face in an image including an operator. [Figure 8] 10 is an explanatory diagram showing the detection result of the direction of the face (the distance between the eyes) in an image including the operator. FIG. [Figure 9] FIG. 10 is an explanatory diagram showing the detection result of the body orientation (distance between both shoulders) in an image including an operator. [Figure 10] 10 is an explanatory diagram showing the detection result of the eye opening degree (aspect ratio) in an image including an operator. FIG. [Figure 11] 10 is a flowchart illustrating a process (monitoring process) of a control unit of the information processing device. [Figure 12] 10 is a flowchart illustrating processing (calibration processing when an operator makes a change) by a control unit of an information processing device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] (Embodiment 1) Hereinafter, embodiments will be described with reference to the drawings. FIG. 1 is a schematic diagram illustrating the configuration of a camera unit 1 including an information processing device 2 (camera ECU) according to the first embodiment. FIG. 2 is a block diagram illustrating the physical configuration of the information processing device 2. The camera unit 1 includes a camera 11 mounted on, for example, an OHC (overhead console) in a vehicle C, and the information processing device 2. The information processing device 2 performs image processing on an image captured by the camera 11 and uses the image processing results to perform monitoring of an operator operating the vehicle C. Such a camera unit 1 may function as, for example, a cabin monitor system.

[0009] The camera 11 is configured, for example, with a CMOS camera, and is arranged, for example, on an OHC (overhead console). The number of cameras 11 according to this embodiment may be any number greater than one. The camera 11 is communicably connected to the information processing device 2, for example, via a serial cable, and outputs captured images such as moving images to the information processing device 2 periodically or in real time. Note that "real time" in the various processes in this specification does not mean strictly immediate processing or simultaneous processing, but means that processing is performed as quickly as possible.

[0010] The information processing device 2 includes a control unit 20, a storage unit 23, an input / output I / F 21, and a communication unit 22, and functions as a camera ECU that performs image processing on images captured by the camera 11 and performs processing according to the results of the image processing. The control unit 20 is configured with a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), etc., and performs various control processes, arithmetic processes, etc. by reading and executing a control program P (program product) and data stored in advance in the storage unit 23.

[0011] The storage unit 23 is configured by a volatile memory element such as a RAM (Random Access Memory) or a non-volatile memory element such as a ROM (Read Only Memory), an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The storage unit 23 pre-stores a control program P and data that the control unit 20 references when performing various calculations. Furthermore, the storage unit 23 may also store entity files of learning models such as a face detection model 201, a face landmark model 202, and a posture detection model 203. These learning models will be described later.

[0012] Alternatively, learning models such as the face detection model 201, the face landmark model 202, and the posture detection model 203 may be implemented in an AI chip, and the control unit 20 may input images to these learning models and obtain recognition results for objects from the learning models by performing input / output processing on the AI ​​chip via an internal bus or the like.

[0013] The input / output I / F 21 is, for example, a communication interface for serial communication. The information processing device 2 is communicably connected to the camera 11 and the like via the input / output I / F 21. Furthermore, a speaker or a display device having an audio output function may be connected to the input / output I / F 21. A warning sound or the like is emitted from the speaker or the like based on the result of the monitoring process executed by the information processing device 2.

[0014] The communication unit 22 is an input / output interface that uses a communication protocol such as CAN or Ethernet (registered trademark). The control unit 20 may communicate with other ECUs 3 connected to the in-vehicle network 4 via the communication unit 22. For example, as shown in FIGS. 1 and 2 , the information processing device 2 according to this embodiment communicates with the ECUs 3 via the in-vehicle network 4 and acquires various communication data (CAN messages, Ethernet frames, etc.) from these ECUs 3.

[0015] FIG. 3 is an explanatory diagram regarding the generation process of the face detection model 201. The face detection model 201 is implemented in, for example, an AI chip or the like. Alternatively, the face detection model 201 may be configured as a software functional unit controlled by the control unit 20. In this embodiment, the information processing device 2 generates the face detection model 201. However, the present invention is not limited to this. The face detection model 201 may be generated by a model generation device such as a server device. The control unit 20 of the information processing device 2 may train a pre-learning machine learning model (for example, a neural network such as YOLO or R-CNN) using existing training data or training data prepared for the face detection model 201, thereby generating the face detection model 201 that recognizes (detects) an object (the operator's face) included in an input image (an image including an operator) when the input image is received.

[0016] The object recognition result output by the face detection model 201 includes at least whether or not an object (face) was recognized. If an object is recognized, the recognition result includes the area (bounding box) and type (class) of the object in the image. The recognition result may also include a confidence score (predicted probability of each class) of the recognized object.

[0017] The training data includes question data and answer data, and an image (an image including the operator) acquired from the camera 11 corresponds to the question data, and the recognition result for the object (face) corresponds to the answer data. The area and type of the object (face), which is the answer data, may be set by annotating (adding) the image. The data set of question data and answer data included in the training data for learning the face detection model 201 and the data set of input data and output data when the face detection model 201 is used are synonymous, and if defined in one data set, it naturally applies to the other data set as well.

[0018] The neural network (face detection model 201) trained using training data is expected to be used as a program module that is part of artificial intelligence software. The face detection model 201 is used in the information processing device 2 that includes the control unit 20 (CPU, etc.) and storage unit 23 as described above, and a neural network system is configured by being executed by the information processing device 2 that has such calculation processing capabilities. That is, the control unit 20 of the information processing device 2 performs calculations to extract feature amounts of an image input to an input layer in accordance with instructions from the face detection model 201 stored in the storage unit 23, and outputs a recognition result for the object.

[0019] The face detection model 201 is configured using, for example, YOLO or R-CNN (Region Convolutional Neural Network), and includes an input layer that accepts image input, a middle layer that extracts features from the image, and an output layer that outputs recognition results for the object. The input layer has multiple neurons that accept image input and passes the input values ​​to the middle layer. The middle layer is defined using an activation function such as a ReLU function or a sigmoid function, has multiple neurons that extract features for each input value, and passes the extracted features to the output layer. Parameters such as weighting coefficients and bias values ​​of the activation function are optimized using backpropagation. The output layer is configured, for example, by a fully connected layer and outputs recognition results for the object based on the features output from the middle layer. The output layer may also include, for example, a softmax layer, which outputs the probability (confidence score) of the object.

[0020] In this embodiment, the face detection model 201 is R-CNN or the like, but is not limited thereto. The face detection model 201 may be constructed using other machine learning algorithms, such as neural networks other than R-CNN, transformers, BERT, GPT, recurrent neural networks (RNNs), long-short term models (LSTMs), support vector machines (SVMs), Bayesian networks, linear regression, regression trees, multiple regression, random forests, and ensembles. Alternatively, the face detection model 201 may be constructed using an artificial intelligence chatbot based on an LLM, such as ChatGPT. In this case, ChatGPT may be fine-tuned to efficiently output recognition results for the target object. Alternatively, a question (prompt) generated using an external database, such as a WebDB, may be input together with an image to a prompt input I / F, which is an input interface of ChatGPT.

[0021] FIG. 4 is an explanatory diagram regarding the generation process of the facial landmark model 202. Like the face detection model 201, the facial landmark model 202 is implemented on an AI chip or the like, or configured as a software function unit controlled by the control unit 20. Like the face detection model 201, the facial landmark model 202 may also be configured using YOLO, R-CNN, or the like. The training data includes question data and answer data, where an image acquired from the camera 11 (an image including the operator) corresponds to the question data, and the areas or positions of landmarks indicating characteristic features on the face, such as the eyes and nose, and attributes indicating which features on the face the landmarks indicate correspond to the answer data. The control unit 20 of the information processing device 2 may train a pre-learning machine learning model (for example, a neural network such as YOLO or R-CNN) using existing training data or training data prepared for the facial landmark model 202, thereby generating the facial landmark model 202 that recognizes (detects) objects (facial landmarks) included in an input image (an image including the operator) by inputting the image.

[0022] FIG. 5 is an explanatory diagram illustrating a process for generating the posture detection model 203. Like the face detection model 201, the posture detection model 203 is implemented on an AI chip or the like, or configured as a software function unit controlled by the control unit 20. Like the face detection model 201, the posture detection model 203 may be configured using YOLO, R-CNN, or the like. The training data includes question data and answer data. Images (images including the operator) acquired from the camera 11 correspond to the question data, and the areas or positions of landmarks indicating characteristic parts of the upper body, such as the shoulders, elbows, neck, and waist, and attributes indicating which parts of the human body the landmarks indicate correspond to the answer data. The control unit 20 of the information processing device 2 may train a pre-learning machine learning model (e.g., a neural network such as YOLO or R-CNN) using existing training data or training data prepared for the posture detection model 203, thereby generating the posture detection model 203 that recognizes (detects) objects (landmarks in the upper body) included in an input image (an image including the operator) by inputting the image.

[0023] 6 is a flowchart illustrating processing (calibration processing) of the control unit 20 of the information processing device 2. The control unit 20 of the information processing device 2 performs the following processing, for example, when the vehicle C is in an activated state. While performing the following processing, the control unit 20 of the information processing device 2 steadily or periodically acquires moving or still images of the operator from the camera 11, and stores the acquired images in the storage unit 23 with timestamps indicating the capture times of the images. By storing each acquired image (video frame) in this manner in association with the capture time of the image, it is possible to store (save) images captured at multiple consecutive time points in chronological order in the storage unit 23.

[0024] The control unit 20 of the information processing device 2 determines whether the vehicle C has started to move from a stopped state (S101). The control unit 20 of the information processing device 2 periodically acquires a message including the vehicle speed and the like from the ECU 3, which periodically outputs the message via the in-vehicle network 4, for example. Based on the acquired message including the vehicle speed and the like, the control unit 20 of the information processing device 2 determines whether the vehicle C has started to move from a stopped state depending on whether the vehicle C has started to move after stopping, i.e., after the vehicle speed has become substantially 0 km / h, i.e., whether the vehicle speed has become greater than 0 km / h. Alternatively, the control unit 20 of the information processing device 2 may determine that the vehicle C has started to move from a stopped state when it acquires a message including an item indicating that the vehicle C has started to move (a running start message) from the ECU 3, which outputs the message when the vehicle C starts to move from a stopped state.

[0025] When the vehicle C starts traveling from a stopped state (S101: YES), the control unit 20 of the information processing device 2 acquires the vehicle speed and steering angle (S102). The control unit 20 of the information processing device 2 periodically acquires messages including the vehicle speed and the like from the ECU 3 that manages the vehicle speed via the in-vehicle network 4. Furthermore, the control unit 20 of the information processing device 2 periodically acquires messages including the steering angle and the like from the ECU 3 that manages the steering angle via the in-vehicle network 4. The control unit 20 of the information processing device 2 associates the acquired vehicle speed and steering angle with the acquisition time points at which the messages including the vehicle speed or steering angle were acquired, and stores them in the storage unit 23. By storing the vehicle speed and steering angle in association with the acquisition time points in this manner, the values ​​of the vehicle speed and steering angle acquired at consecutive times can be stored (saved) in the storage unit 23 as history data arranged in chronological order.

[0026] The control unit 20 of the information processing device 2 determines whether the vehicle C is traveling straight based on the vehicle speed and the steering angle (S103). Based on the periodically acquired messages including the vehicle speed and the like and the messages including the steering angle and the like, the control unit 20 determines that the vehicle C is traveling straight based on, for example, the vehicle speed being equal to or greater than a predetermined speed and the steering angle being within a predetermined angle. The predetermined speed is, for example, 5 km / h, and the predetermined angle is, for example, ±5° when straight traveling is set as the reference angle. These predetermined values ​​or thresholds, such as the predetermined speed and the predetermined angle, used by the control unit 20 to perform various calculations are stored in advance in the storage unit 23.

[0027] If the vehicle C is in a straight-ahead state (S103: YES), the control unit 20 of the information processing device 2 determines whether the straight-ahead state has continued for a predetermined period or more (S104). When the control unit 20 of the information processing device 2 determines that the vehicle C is in a predetermined straight-ahead state (vehicle speed is equal to or greater than a predetermined speed and steering angle is within a predetermined angle), it determines whether the straight-ahead state has continued (maintained) for a predetermined period, for example, 5 seconds.

[0028] If the straight-travel state continues for a predetermined period or more (S104: YES), the control unit 20 of the information processing device 2 generates reference data (S105). If the straight-travel state continues for a predetermined period or more, the control unit 20 of the information processing device 2 identifies the start and end points of the predetermined period. The control unit 20 of the information processing device 2 identifies the start and end points according to the acquisition points associated with the vehicle speed and steering angle used when determining whether the straight-travel state has continued for a predetermined period or more. In this case, the control unit 20 of the information processing device 2 may identify the start point as the point at which it is determined in the processing of S103 that the vehicle C is in a straight-travel state based on the vehicle speed and steering angle, and the end point as the point at which it is determined that the straight-travel state has continued for a predetermined period or more.

[0029] The control unit 20 of the information processing device 2 acquires images captured during the specified period according to the start and end points of the specified period. The storage unit 23 stores images acquired from the camera 11 in association with the image capture time, and the control unit 20 of the information processing device 2 acquires images captured during the specified period during which the straight-ahead state continued, i.e., multiple images captured from the start to the end of the specified period. If the images are a moving image, the multiple images correspond to multiple frames (frame group) that make up the moving image.

[0030] The control unit 20 of the information processing device 2 extracts the face position, face direction, body direction, or eye opening degree of the operator contained in each of a plurality of images (frame group) captured over a predetermined period of time. The control unit 20 of the information processing device 2 calculates the average value, median value, or mode value of each of the face positions extracted in each of the images, and generates the average value as reference data indicating the face position (face position reference data).

[0031] The control unit 20 of the information processing device 2 calculates the average, median, or mode of each of the facial orientations extracted from each of these images and generates the average as reference data indicating the facial orientation (reference data for facial orientation). The control unit 20 of the information processing device 2 calculates the average, median, or mode of each of the body orientations extracted from each of these images and generates the average as reference data indicating the body orientation (reference data for body orientation). The control unit 20 of the information processing device 2 calculates the maximum value of the eye opening degree extracted from each of these images and generates the maximum value as reference data indicating the eye opening degree (reference data for eye opening degree). The control unit 20 of the information processing device 2 may generate reference data for all of the operator's facial position, facial orientation, body orientation, and eye opening degree, or may generate reference data for one or more of them. In other words, the control unit 20 of the information processing device 2 may generate reference data for at least one of the operator's facial position, facial orientation, body orientation, and eye opening degree depending on the type of monitoring process to be executed. Below, the reference data for the operator's face position, face direction, body direction, and eye opening will be described.

[0032] FIG. 7 is an explanatory diagram showing the detection result of the position (bounding box) of a face in an image including an operator. The control unit 20 of the information processing device 2 inputs each of a plurality of images (frame groups) into a face detection model 201 to detect the position of the face in each of the images. The face detection model 201 is, for example, a FaceDetect model, and when an image including an operator is input, it detects the face of the operator and outputs a rectangular frame (bounding box) indicating the position of the detected face, i.e., the area of ​​the face in the image, superimposed on the image. The bounding box is defined by the coordinates of, for example, the upper left vertex and the lower right vertex in the coordinate system of the image.

[0033] The control unit 20 of the information processing device 2 extracts a bounding box from each of multiple images (frame group) captured within a predetermined period while the vehicle C is traveling straight, and derives a bounding box that is the average value of the extracted multiple bounding boxes. In deriving the average value of the bounding boxes, the control unit 20 of the information processing device 2 may derive the coordinate values ​​of the upper left and lower right vertices of the bounding box that is the average value by calculating the average value of the coordinate values ​​of the upper left and lower right vertices of each bounding box. Alternatively, the control unit 20 of the information processing device 2 may calculate the center of gravity or center point of each bounding box, and then calculate the average value of the X and Y coordinate values ​​of each calculated center of gravity, etc., to derive the coordinates of the center of gravity of the bounding box that is the average value.

[0034] The control unit 20 of the information processing device 2 stores the calculated average value of the bounding boxes as reference data (reference data for the position of a face) in the storage unit 23. Alternatively, the control unit 20 of the information processing device 2 may set an area obtained by expanding the calculated average value of the bounding boxes by, for example, 25% as the reference data (frame for detection). In this embodiment, the expanded area is used as the reference data (reference data for the position of a face). When storing the generated reference data in the storage unit 23, the control unit 20 of the information processing device 2 may store the data by adding a timestamp or the like indicating the time of generation.

[0035] The reference data (face position reference data) set in the expanded area by 25% or the like in this way is used in the monitoring process for the operator. The control unit 20 of the information processing device 2 detects the position of the operator's face in an image including the operator acquired during the monitoring process using the face detection model 201, and may determine that the operator is not facing forward if the detected face position deviates from the area indicated by the reference data by, for example, 25% or more, and may issue a warning sound or the like if the state where the face deviates by, for example, 70% or more continues for three seconds, indicating that the operator is drowsy at the wheel.

[0036] FIG. 8 is an explanatory diagram showing the detection result of the face orientation (distance between the eyes) in an image including an operator. The control unit 20 of the information processing device 2 inputs each of a plurality of images (frame groups) to the face landmark model 202, thereby detecting the face orientation in each of the images. Alternatively, the control unit 20 of the information processing device 2 may, when outputting an image including an operator to the face detection model 201, extract an image (face area image) of the face area (bounding box) detected by the face detection model 201 and input the extracted face area image to the face landmark model 202. The face landmark model 202 is, for example, a Face Landmark model or a Mediapipe model. When an image including a face is input, the face landmark model 202 detects multiple landmarks that are characteristic features of the face, such as the eyes, nose, and mouth, and outputs the points indicated by these landmarks superimposed on the image. The detected landmarks are assigned coordinate values ​​in an image coordinate system and attributes indicating each facial feature (eyes, nose, mouth, etc.).

[0037] The control unit 20 of the information processing device 2 identifies two landmarks representing the left and right eyes, and calculates the distance between the two identified landmarks (landmarks representing the eyes) in the image coordinate system, thereby calculating the distance between the left and right eyes. The control unit 20 of the information processing device 2 calculates the average value of the distances between the eyes calculated for each of a plurality of images (frame groups), and derives the average value as reference data (reference data for face direction).

[0038] The reference data (face direction reference data) set based on the average value of the distance between the eyes thus derived is used in the monitoring process for the operator. The control unit 20 of the information processing device 2 detects the distance between the eyes of the operator in an image including the operator acquired during the monitoring process, using the face landmark model 202. The control unit 20 of the information processing device 2 may determine that the operator is looking left if the detected distance between the eyes is longer than the reference data, and may determine that the operator is looking right if the detected distance between the eyes is shorter than the reference data. If the operator continues to look left or right for a predetermined period of time, such as three seconds, the control unit 20 may issue a warning sound or the like indicating that the operator is inattentive driving.

[0039] 9 is an explanatory diagram showing the detection result of the body orientation (distance between both shoulders) in an image including an operator. The control unit 20 of the information processing device 2 inputs each of a plurality of images (frame groups) into a posture detection model 203, thereby detecting the posture of the operator in each of the images. The posture detection model 203 is, for example, a posedetect model, and when an image including an operator is input, it detects characteristic parts of the operator's upper body, such as joints such as the shoulders, and outputs points indicating each of these parts by superimposing them on the image. These detected parts are assigned coordinate values ​​in the image coordinate system and attributes indicating each part of the body (shoulders, elbows, wrists, neck, waist, etc.).

[0040] The control unit 20 of the information processing device 2, for example, identifies two parts indicating the left and right shoulders and calculates the distance between the identified two parts (both shoulders) in the image coordinate system, thereby calculating the distance between the left and right shoulders. The control unit 20 of the information processing device 2 calculates the average value of the distances between the both shoulders calculated for each of the multiple images (frame groups), and derives the average value as reference data (reference data for body orientation).

[0041] The reference data (body orientation reference data) set based on the average value of the distance between the shoulders thus derived is used in the monitoring process for the operator. The control unit 20 of the information processing device 2 detects the distance between the shoulders of the operator in an image including the operator acquired during the monitoring process, using the posture detection model 203. The control unit 20 of the information processing device 2 may determine that the operator is looking left if the detected distance between the shoulders is longer than the reference data, or determine that the operator is looking right if the detected distance between the shoulders is shorter than the reference data, and may issue a warning sound or the like to indicate that the operator is in a distracted driving state if the operator continues to look to the left or right for a predetermined period of time, such as three seconds.

[0042] FIG. 10 is an explanatory diagram showing the detection result of the eye opening degree (aspect ratio) in an image including an operator. The control unit 20 of the information processing device 2 inputs each of a plurality of images (frame groups) to the face landmark model 202, thereby detecting the eye opening degree in each of the images. Alternatively, the control unit 20 of the information processing device 2 may, when outputting an image including an operator to the face detection model 201, extract an image (face area image) of the face area (bounding box) detected by the face detection model 201, and input the extracted face area image to the face landmark model 202. The face landmark model 202 is, for example, a Face Landmark model or a Mediapipe model. When an image including a face is input, the face landmark model 202 detects multiple landmarks that are characteristic parts of the face, such as the eyes, nose, and mouth, and outputs the points indicated by these landmarks superimposed on the image. In this embodiment, the face landmark model 202 detects multiple landmarks (P1, P2, P3, P4, P5, P6) around or along the outline of the eyes. These detected landmarks are assigned attributes indicating coordinate values ​​in the image coordinate system and the names of the parts of the eye (upper eyelid, lower eyelid, etc.).

[0043] The control unit 20 of the information processing device 2 calculates the eye length (horizontal length) and the distance from the upper eyelid to the lower eyelid (vertical length: eye height) from the detected multiple landmarks (P1, P2, P3, P4, P5, P6), and calculates the eye aspect ratio (vertical eye length / horizontal eye length) by dividing the vertical eye length by the horizontal eye length, as data related to the eye opening. In this embodiment, the distance from the upper eyelid to the lower eyelid (vertical length) is calculated by adding up the distances at two locations (P2-P6, P3-P5), so the eye length (horizontal length) is divided by double the value: {eye height (distance from P2 to P6) + (distance from P3 to P5)} / eye length (distance from P1 to P4) × 2}. This allows for the calculation of the eye opening (EAR: Eye Aspect Ratio), which indicates the ratio between "eye height" and "eye length." The control unit 20 of the information processing device 2 derives the maximum value of the eye opening degree (aspect ratio) calculated for each of the plurality of images (frame group) as reference data (eye opening degree reference data).

[0044] The reference data (eye opening reference data) set based on the maximum value of the eye opening (aspect ratio) derived in this manner is used in the monitoring process for the operator. The control unit 20 of the information processing device 2 detects the eye opening (aspect ratio) of the operator in an image including the operator acquired during the monitoring process using the face landmark model 202. The control unit 20 of the information processing device 2 compares the detected eye opening with the reference data, and may issue a warning sound or the like if, for example, the detected eye opening is less than 90% of the reference data (maximum eye opening). The control unit 20 of the information processing device 2 may perform the monitoring process for either the left or right eye that has been recognized.

[0045] Furthermore, the control unit 20 of the information processing device 2 may vary the reference data (reference data for eye opening) depending on the face direction (upward or downward). The control unit 20 of the information processing device 2 may determine whether the face direction of the operator is upward or downward based on the landmarks detected by the face landmark model 202.

[0046] The control unit 20 of the information processing device 2 may decrease the reference data (reference data for eye opening) by, for example, 10% when the operator's face is facing downward. When the operator looks down, the eyes appear to be less open, which may lead to a concern that the eyes are more likely to be determined to be closed. By decreasing the reference data (reference data for eye opening) (changing the threshold to a stricter value), it is possible to prevent erroneous determinations from occurring. The control unit 20 of the information processing device 2 may increase the reference data (reference data for eye opening) by, for example, 10% when the operator's face is facing upward. When the operator looks up, the eyes appear to be more open, which may lead to a concern that the eyes are more likely to be determined to be closed. By increasing the reference data (reference data for eye opening) (changing the threshold to a looser value), it is possible to prevent erroneous determinations from occurring.

[0047] If the current generation of reference data is the first generation since the vehicle C was started, the generated reference data is stored as the initial reference data in the storage unit 23. If reference data has already been generated and stored in the storage unit 23 and is being applied to the monitoring process currently being executed, the control unit 20 of the information processing device 2 applies the currently generated reference data instead of the currently applied reference data, that is, overwrites the current reference data stored in the storage unit 23 with the currently generated reference data, thereby updating the reference data used in the monitoring process.

[0048] In the present embodiment, the control unit 20 of the information processing device 2 generates the reference data, but this is not limiting. The reference data may be generated by an external server such as a cloud server wirelessly connected to the information processing device 2. In this case, the information processing device 2 may output an image including the operator to the external server via an external network such as a TCU (Telematics Control Unit) mounted on the vehicle C and the Internet, and instruct the external server to generate the reference data. The external server generates the reference data based on the image from the information processing device 2 and outputs the generated reference data to the information processing device 2. The control unit 20 of the information processing device 2 may generate the reference data by acquiring the reference data from the external server.

[0049] If the vehicle C is not traveling straight (S103: NO), or if the straight-travel state has not continued for a predetermined period or longer (S104: NO), the control unit 20 of the information processing device 2 does not generate reference data and continues to use the currently applied reference data (S1031). If the vehicle C is not traveling straight or if the straight-travel state has not continued for a predetermined period or longer, it is difficult for the operator to imagine that the vehicle is facing forward, so the control unit 20 of the information processing device 2 does not generate reference data and continues to use the currently applied reference data.

[0050] If the vehicle C has not started traveling from a stopped state (S101: NO), after execution of S105 or S1031, the control unit 20 of the information processing device 2 executes a monitoring process for the operator using the reference data (S106). If the vehicle C has not started traveling from a stopped state, that is, while traveling after generating reference data by maintaining an initial straight traveling state for a predetermined period after stopping, the control unit 20 of the information processing device 2 executes the monitoring process by continuously applying the reference data. If the reference data has not been generated when the vehicle C has started traveling from a stopped state, the control unit 20 of the information processing device 2 may execute the monitoring process using, for example, a preset initial value (default value).

[0051] As described above, the control unit 20 of the information processing device 2 executes a monitoring process corresponding to the type of generated reference data (face position reference data, face orientation reference data, body orientation reference data, eye opening reference data) in accordance with the type of the reference data. Furthermore, the control unit 20 of the information processing device 2 may execute a monitoring process by combining two or more types of reference data.

[0052] 11 is a flowchart illustrating a process (monitoring process) of the control unit 20 of the information processing device 2. The control unit 20 of the information processing device 2 may perform the process shown in the following flowchart as a subroutine of process S106, as an example of the monitoring process.

[0053] The control unit 20 of the information processing device 2 determines whether the operator's eyes are closed based on the acquired image (T101). The control unit 20 of the information processing device 2 detects whether the operator's eyes are closed (Eye Closed) using the face landmark model 202, and derives the detection result as a provisional determination. If it is determined that the operator's eyes are not closed (T101: NO), the control unit 20 of the information processing device 2 executes detection of the operator's blinking or eye opening degree (T1011).

[0054] If it is determined that the eyes of the operator are closed (T101: YES), the control unit 20 of the information processing device 2 determines whether the operator is facing forward (T102). If it is determined that the eyes of the operator are closed, the control unit 20 of the information processing device 2 checks the direction of the face based on the detection result of the face landmark model 202, and determines whether the face is facing forward, that is, whether the face is not looking down extremely or excessively.

[0055] If it is determined that the operator is not facing forward (T102: NO), the control unit 20 of the information processing device 2 determines that the direction of the operator's face is inappropriate (T1022). If it is determined that the operator is not facing forward, that is, if it is determined that the operator is not facing either left or right, the control unit 20 of the information processing device 2 determines that the direction of the operator's face is inappropriate and that the operator is distracted driving.

[0056] When it is determined that the operator is facing forward (T102: YES), the control unit 20 of the information processing device 2 determines whether the operator's eye opening degree is small for a predetermined period of time (T103). When it is determined that the operator is facing forward, the control unit 20 of the information processing device 2 determines whether the operator's eye opening degree (EAR) is small for a predetermined period of time, for example, two seconds. When the operator's eye opening degree is not small for a predetermined period of time (T103: NO), the control unit 20 of the information processing device 2 determines that the operator is not sleeping (T104). When the operator's eye opening degree is small for a predetermined period of time (T103: YES), the control unit 20 of the information processing device 2 generates an alarm (T105).

[0057] The control unit 20 of the information processing device 2 determines whether the degree of eye opening of the operator is small for a predetermined period of time (T106). If the degree of eye opening of the operator is not small for a predetermined period of time (T106: NO), the control unit 20 of the information processing device 2 determines that the operator is not sleeping (T107). If the degree of eye opening of the operator is small for a predetermined period of time (T106: YES), the control unit 20 of the information processing device 2 determines that the operator is sleeping (T1061). After the sound warning, if the degree of eye opening (EAR) of the operator continues to be small, the control unit 20 of the information processing device 2 determines that the operator is sleeping, and if it is large, the control unit 20 determines that the operator is not sleeping.

[0058] By performing such processing, the control unit 20 of the information processing device 2 can efficiently determine whether or not the operator is actually asleep by combining the shaking of the face and posture with steering angle information from the vehicle C. That is, even if the face is facing forward but the eyes are looking down, it can be prevented from determining that the operator is not asleep but has their eyes closed, and issuing a warning as a drowsy driver detection after a predetermined period of time has passed.

[0059] The control unit 20 of the information processing device 2 determines whether the vehicle C has been stopped (S107). The control unit 20 of the information processing device 2 periodically acquires a message including data indicating whether the ignition switch is on (started) or off (stopped) from, for example, the ECU 3 that manages the ignition switch, and determines whether the vehicle C has been stopped (ignition switch: off) by referring to the acquired message. If it is determined that the vehicle C has not been stopped (S107: NO), the control unit 20 of the information processing device 2 performs loop processing to execute the processing from S101 again. By performing such loop processing, the control unit 20 of the information processing device 2 can repeat the processing of generating and updating the reference data when the vehicle C starts running (ignition switch: on) and travels straight for the first time after starting to drive.

[0060] When the vehicle C is stopped (S107: YES), the control unit 20 of the information processing device 2 erases the reference data stored in the memory unit 23 (S108). When the vehicle C is stopped (ignition switch: OFF), the control unit 20 of the information processing device 2 may erase the reference data stored in the memory unit 23 or initialize the reference data.

[0061] According to this embodiment, the information processing device 2 continuously or periodically acquires images including an operator operating the vehicle C using a camera 11 mounted on the vehicle C. The images may be still images or moving images captured at a predetermined frame rate. The information processing device 2 further acquires information regarding the driving state of the vehicle C (vehicle C state information) based on, for example, CAN messages acquired from various ECUs 3 communicably connected via the in-vehicle network 4. The vehicle C state information includes, for example, vehicle speed and steering angle (angle of the steering shaft: steering angle). Based on the acquired vehicle C state information, the information processing device 2 can recognize the driving state determined by the vehicle speed and steering angle. When the information processing device 2 determines that the driving state of the vehicle C is a straight-line driving state at a predetermined speed (e.g., 5 km / h) or higher and that the straight-line driving state has continued for a predetermined period (e.g., 5 seconds) or longer, the information processing device 2 generates reference data for performing monitoring processing of the operator based on images captured during the predetermined period, i.e., the period during which the straight-line driving state continued. Regarding whether or not the vehicle C is in a straight-ahead state, the vehicle may be determined to be in a straight-ahead state if the steering angle is within a predetermined angle range (e.g., ±5° when straight-ahead driving is set as the reference angle of 0°). Parameters (determination parameters), such as a predetermined speed (vehicle speed), a predetermined angle range (steering angle), and a predetermined period (state duration) for determining whether the vehicle C is in a straight-ahead state, may be variably set by, for example, an input operation by the operator. In this way, the information processing device 2 generates reference data for performing monitoring processing based on images of the operator captured during a period in which the vehicle C maintains a straight-ahead state, which is a predetermined driving state based on the vehicle speed and steering angle, after the vehicle C starts traveling. This eliminates the need for a process to pre-register the reference data. That is, when performing monitoring processing to detect, for example, distracted driving or drowsy driving using an image of the operator of the vehicle C captured by the camera 11 mounted on the vehicle C, user registration using an image of the operator's face or upper body may be required. However, in this embodiment, pre-user registration is not required.Therefore, even if there are regulations prohibiting facial recognition using an image including the operator's face from the perspective of personal identification, by performing the processing according to this embodiment, it is possible to generate reference data and execute monitoring processing using the reference data without performing user registration requiring facial recognition. The driving state predetermined by the vehicle speed and steering angle substantially corresponds to the straight-ahead state of the vehicle C, and it is assumed that the operator tends to face forward while the straight-ahead state continues. Therefore, it is assumed that, during a period in which the predetermined driving state (straight-ahead state) continues, the frames (frame group) included in a video capturing the operator will be dominated by frames in which the operator faces forward. Since the reference data is generated using multiple images (frame group constituting the video) in which the operator faces forward, the reference data corresponds to data when the operator faces forward. In monitoring processes targeting distracted driving or drowsy driving, an alert is generated if the direction of the operator's face or the direction of the upper body or other body parts deviates from the forward direction to the left or right. Since the reference data corresponds to data when the operator is facing forward, the accuracy of the monitoring process can be guaranteed or improved.

[0062] According to this embodiment, the information processing device 2 determines whether the vehicle C is in a predetermined driving state when it first travels after stopping. That is, the information processing device 2 determines that the vehicle C is in a predetermined driving state when it first travels straight after starting to travel. Therefore, after the information processing device 2 determines that the vehicle C is in a predetermined driving state (straight driving state) and generates reference data when the vehicle C first travels after stopping, the information processing device 2 continues to apply the generated reference data without making the determination until the vehicle C next stops. When the vehicle C is traveling, a driving state in which the vehicle is traveling at a predetermined speed or higher and the steering angle is within a predetermined angle range, i.e., a state in which the straight driving state continues for a predetermined period or more, may occur multiple times on a regular basis. In contrast, the information processing device 2 determines whether the vehicle C is in the predetermined driving state (straight driving state) when it first travels after stopping, thereby preventing the determination from being made excessively.

[0063] According to this embodiment, the reference data generated by the information processing device 2 is data obtained based on at least one of the face position, face orientation, and body orientation of the operator included in images (frame group) acquired during a predetermined period, i.e., a period during which the vehicle C maintains a straight-ahead state (a predetermined driving state). That is, the reference data may be data obtained based on the face position, face orientation, or body orientation of the operator included in the images, or a combination of these data. When generating the reference data, the information processing device 2 may extract regions of body parts, such as the face or upper body, from the images captured during the predetermined period using an object detection model configured, for example, by R-CNN or YOLO, or using a face detection model 201 (Facedetect), a posture detection model 203 (Poseddetect), or a posture estimation model (OpenPose), which are widely used as general-purpose learning models, and use the extracted regions as reference data. In this case, the face area is indicated by a bounding box output by the object detection model, and an area obtained by expanding this area (bounding box) by, for example, 25% may be set as reference data (detection frame). Alternatively, when generating the reference data, the information processing device 2 may calculate the distance between the eyes on the operator's face and set the distance between the eyes as reference data indicating the orientation of the face. Alternatively, when generating the reference data, the information processing device 2 may calculate the distance between the shoulders on the operator's upper body and set the distance between the shoulders as reference data indicating the orientation of the body. In this way, by generating reference data using the position, orientation, or orientation of the operator's face, it is possible to ensure or improve the accuracy of the monitoring process.

[0064] According to this embodiment, images captured over a predetermined period of time are composed of a frame group including multiple frames (still images) arranged in chronological order according to the frame rate. That is, the images captured over the predetermined period of time include multiple images. The information processing device 2 may calculate data indicating the position, orientation, or body orientation of the face for each of the multiple images captured over the predetermined period of time, and derive the average value of this calculated data as reference data. When generating reference data based on the face position, the information processing device 2 may average the coordinates of bounding boxes obtained by object detection targeting the face for each of multiple frames (still images) included in a video captured over the predetermined period of time to calculate an area indicating the position of the face when the operator is facing forward. Alternatively, when generating reference data based on the face orientation, the information processing device 2 may average the distance between the eyes of the face for each of multiple frames (still images) included in a video captured over the predetermined period of time to calculate the distance between the eyes when the operator is facing forward. Alternatively, when generating reference data based on the body orientation, the information processing device 2 may average the distance between the shoulders of the upper body in each of multiple frames (still images) included in a video captured within a predetermined period to obtain the distance between the shoulders when the operator is facing forward. While the averaging process is used to generate the reference data, this is not limited thereto. The information processing device 2 may generate the reference data using the median or mode of data (bounding box coordinates, distance between the eyes, distance between the shoulders) derived based on each of multiple frames (still images) included in a video captured within a predetermined period. In this way, even if there is some bias in the position, orientation, or orientation of the operator's face in each of multiple frames (still images) included in a video captured within a predetermined period, the influence of the bias can be mitigated by performing a process such as averaging, and reference data for when the operator is facing forward can be suitably generated.

[0065] According to this embodiment, the reference data generated by the information processing device 2 is data obtained based on the degree of eye opening of the operator included in images (frame group) acquired during a predetermined period, i.e., a period during which the vehicle C maintains a straight-ahead state (a predetermined driving state) and the size of the operator's eyes when the eyes are wide open. The information processing device 2 extracts a face area image showing the area of ​​the operator's face included in the acquired images and inputs the face area image into a face landmark model 202 (e.g., MediaPope) that detects each part of the face, thereby detecting the upper eyelid, lower eyelid, and points on the left and right sides of the eye, which are the peripheral areas of the eye. The information processing device 2 calculates the horizontal length of the eye based on the distance between the detected left and right points of the eye, and calculates the vertical length of the eye based on the distance between the upper eyelid and the lower eyelid. The information processing device 2 may then derive the aspect ratio of the eye (vertical length of the eye / horizontal length of the eye) calculated by dividing the vertical length of the eye by the horizontal length of the eye, as data related to the degree of eye opening. In this case, data relating to the degree of eye opening (size) (eye aspect ratio) is calculated for each image (frame group) acquired while the vehicle C maintains a straight-ahead state (predetermined driving state), and the information processing device 2 derives the maximum value of the eye aspect ratio calculated for each of the images (frames) as reference data. Alternatively, the information processing device 2 may generate the reference data using the average, median, or mode of the eye aspect ratio calculated for each of the images (frames). In this way, when the eye aspect ratio is used as data relating to the degree of eye opening (size) of the operator, the maximum value of the aspect ratio is applied as reference data when performing a monitoring process on the operator, thereby making it possible to efficiently detect, for example, drowsy driving based on the degree of eye opening (size) of the operator.

[0066] (Embodiment 2) 12 is a flowchart illustrating the processing (calibration processing when an operator makes a change) of the control unit 20 of the information processing device 2 according to the second embodiment. The control unit 20 of the information processing device 2 performs the following processing, for example, when the vehicle C is in a startup state. The control unit 20 of the information processing device 2 performs the processing from S201 to S204 and S2031, similar to the processing from S101 to S104 and S1031 in the first embodiment.

[0067] The control unit 20 of the information processing device 2 determines whether the operator has changed (S205). Based on the image including the operator used in the previous generation of the reference data, i.e., the currently applied reference data, and the image including the operator acquired in the current process, the control unit 20 of the information processing device 2 determines whether the operator used in the previous generation of the reference data and the operator targeted in the current process are different, i.e., whether the operator has changed (been replaced).

[0068] The control unit 20 of the information processing device 2 stores images periodically or steadily acquired from the camera 11 in the storage unit 23 in association with the time at which the image was captured, and identifies the image used when the previous reference data was generated based on the time at which the image was captured. The time at which the reference data was generated is also stored in the storage unit 23, and the reference data is an image captured during a period in which a straight-line driving state continued for a predetermined period or more. In the current process, the control unit 20 of the information processing device 2 identifies an image captured during a period in which a straight-line driving state continued for a predetermined period or more.

[0069] The control unit 20 of the information processing device 2 may derive the similarity between an image including the operator used in the previous processing and an image including the operator to be used in the current processing, for example, using an image similarity model that outputs the similarity between two input images. Then, if the derived similarity is equal to or greater than a threshold indicating the same person, the control unit 20 of the information processing device 2 may determine that the operator has not changed and is the same person. Alternatively, the control unit 20 of the information processing device 2 may calculate image embedding vectors by vectorizing each of the two images, and if the cosine similarity of these image embedding vectors is equal to or greater than a threshold, determine that the operator has not changed and is the same person.

[0070] If it is determined that the operator has changed (S205: YES), the control unit 20 of the information processing device 2 generates reference data (S206). If it is determined that the operator has changed, that is, if the operator has changed when the vehicle C stopped this time, the control unit 20 of the information processing device 2 generates reference data in the same manner as S105 of the first embodiment.

[0071] If it is determined that the operator has not changed (S205: NO), the control unit 20 of the information processing device 2 does not generate reference data and continues to use the currently applied reference data (S2031). If it is determined that the operator has not changed, that is, if the operator was the same person when the vehicle C stopped this time, the control unit 20 of the information processing device 2 does not generate reference data and continues to use the currently applied reference data, as in S1031 of the first embodiment.

[0072] The control unit 20 of the information processing device 2 performs the processes from S208 to S209 in the same manner as the processes from S107 to S108 in the first embodiment.

[0073] According to this embodiment, after the operator gets in and the vehicle C is started (the ignition switch is turned on) until the vehicle is stopped (the ignition switch is turned off), the information processing device 2 generates reference data for the operator without performing image-based facial recognition each time it determines that the vehicle is in a predetermined driving state (the initial straight-line driving state is maintained for a predetermined period after stopping). In this case, it is assumed that the operator may change (the driver may be changed) while the vehicle C is running. That is, if the operator (driver) is changed while the vehicle C is stopped and idling, the information processing device 2 generates and updates reference data based on an image including the new operator and continues the monitoring process. In contrast, even if the information processing device 2 determines that the vehicle is in a predetermined driving state, if the operator has not changed between the previous time the vehicle was determined to be in a driving state and the time the vehicle is currently determined to be in a driving state, that is, if the operator is the same person, the information processing device 2 continues to use the currently applied reference data without generating reference data based on an image including the operator acquired in the current determination. That is, the information processing device 2 generates reference data based on the image including the operator acquired in the current determination only if the operator has changed between the time when the vehicle was previously determined to be in a predetermined driving state and the time when the vehicle is currently determined to be in a predetermined driving state. When determining whether the operator has changed, the information processing device 2 may compare the image used when the vehicle was previously determined to be in a predetermined driving state with the image used when the vehicle was currently determined to be in a predetermined driving state, and determine that the operator has not changed, i.e., that the operator is the same person, if the degree of coincidence or similarity of the operator included in these images is equal to or greater than a predetermined value. In this way, even if the predetermined driving state (the initial straight driving state after stopping is maintained for a predetermined period of time) occurs multiple times between the start of the vehicle C (ignition switch: on) and the stop of the vehicle C (ignition switch: off), the information processing device 2 does not generate reference data if the operator has not changed, and continues to use the currently applied reference data. This reduces the number of times the reference data is generated and reduces the computational load.

[0074] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0075] Multiple claims may be combined with each other regardless of the form of reference. Multiple dependent claims may be included in the claims, depending on multiple claims. Multiple dependent claims may be included in a multiple dependent claim. If multiple dependent claims are not included in a multiple dependent claim, this does not limit the number of multiple dependent claims that are included in a multiple dependent claim. [Explanation of symbols]

[0076] C vehicle 1 Camera unit (cabin monitor) 11 Camera 2. Information processing unit (camera ECU) 20 Control Unit 21 Input / Output Interface 22 Communications Department 23 Memory section P Control Program (Program Product) 201 Face Detection Model 202 Facial Landmark Model 203 Posture detection model 3 ECU 4. In-vehicle network

Claims

1. Acquiring an image including an operator operating the vehicle using a camera mounted on the vehicle; determining whether the vehicle is in a predetermined running state; When it is determined that the vehicle is in the predetermined traveling state, reference data for performing monitoring processing on the operator is generated based on the acquired image. An information processing method that causes a computer to execute a process.

2. When the vehicle speed is equal to or greater than a predetermined speed and the steering angle of the vehicle is within a predetermined angle range for a predetermined period of time or longer, the vehicle is determined to be in the predetermined running state. The information processing method according to claim 1 .

3. The reference data is data obtained based on at least one of the face position, face direction, and body direction of the operator within the predetermined period. The information processing method according to claim 2 .

4. The reference data is an average value of at least one of the face position, face direction, and body direction of the operator within the predetermined period. The information processing method according to claim 3 .

5. When the vehicle is first driven after being stopped, it is determined whether the vehicle is in the predetermined driving state. The information processing method according to claim 1 .

6. The reference data is data obtained based on data regarding the eye opening degree of the operator within the predetermined period. The information processing method according to claim 2 .

7. The data regarding the eye opening degree of the operator is the maximum value obtained by dividing the maximum vertical length of the operator's eye by the maximum horizontal length of the operator's eye. The information processing method according to claim 6.

8. After generating the reference data, a process of acquiring an image including the operator is continued; The monitoring process for the operator is executed based on the image acquired after the generation of the reference data and the reference data. The information processing method according to any one of claims 1 to 7.

9. Each time it is determined that the vehicle is in the predetermined running state after the operator gets in, the reference data of the operator is generated without performing face authentication based on the image of the operator. The information processing method according to any one of claims 1 to 8.

10. When it is determined that the vehicle is in the predetermined driving state, the image including the operator used when the currently applied reference data was generated is compared with the image including the operator acquired this time to determine whether the operator has been changed; If it is determined that the operator has been changed, based on the image including the operator acquired this time, reference data is generated for performing the monitoring process on the operator; If it is determined that the operator has not been changed, the currently applied reference data is continued to be used without generating reference data based on the image including the operator that has been acquired this time. The information processing method according to any one of claims 1 to 9.

11. Acquiring an image including an operator operating the vehicle using a camera mounted on the vehicle; determining whether the vehicle is in a predetermined running state; When it is determined that the vehicle is in the predetermined traveling state, reference data for performing monitoring processing on the operator is generated based on the acquired image. A program that causes a computer to perform a process.

12. An information processing device mounted on a vehicle and including a control unit, The control unit acquiring an image including an operator operating the vehicle by a camera mounted on the vehicle; determining whether the vehicle is in a predetermined running state; When it is determined that the vehicle is in the predetermined traveling state, reference data for performing monitoring processing on the operator is generated based on the acquired image. Information processing device.

Images