Vehicle control system

Abstract

Improve the technology used to determine a person's posture. [Solution] A vehicle control device 1 comprises a control unit 10 and an imaging unit 12 for imaging the interior of a vehicle, wherein the control unit 19 uses object detection technology to detect an object from an image captured by the imaging unit, calculates a seating likelihood from the image indicating the likelihood that the object's posture is a seated posture, and decides whether to perform a safety posture determination based on the amount of movement of the object based on whether the seating likelihood is equal to or greater than a likelihood threshold, and performs a safety posture determination based on the amount of movement of the object, which includes calculating the amount of movement of the object, and turning on a flag indicating that the object's posture is a safe posture if the amount of movement is less than or equal to a movement threshold, and turning off the flag if the amount of movement exceeds the movement threshold.

Description

Technical Field

[0001] This disclosure relates to a vehicle control device.

Background Art

[0002] Conventionally, techniques for determining a person's posture are known. For example, Patent Document 1 discloses a technique for calculating a likelihood indicating the likelihood that a person's posture is a specific posture (for example, a lying position), and determining that the person's posture is a specific posture when the likelihood is greater than or equal to a threshold value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When using posture determination to judge the permission to start a vehicle, it is desirable to determine that the posture of the occupant in the vehicle is a safe posture, for example, a sitting posture. Seating determination by image recognition is mainly carried out using machine learning. However, if the appearance of the occupant in the image or the installation angle of the camera is inappropriate, the accuracy of seating determination may decrease, and thus the accuracy of safe posture determination may decrease.

[0005] In view of such circumstances, an object of the present disclosure is to improve the technique for determining a person's posture.

Means for Solving the Problems

[0006] A vehicle control device according to an embodiment of the present disclosure includes a control unit and an imaging unit that images the interior of the vehicle, wherein the control unit detects an object from an image captured by the imaging unit using object detection technology, From the aforementioned image, the likelihood of the subject being in a seated position is calculated. Based on whether the seating likelihood is greater than or equal to the likelihood threshold, a decision is made on whether to perform a safety posture determination based on the amount of movement of the subject. Performing a determination based on the amount of movement of the aforementioned target means Calculating the amount of movement of the aforementioned object, and If the amount of movement is less than or equal to the amount of movement threshold, a flag indicating that the target's posture is a safe posture is turned ON; if the amount of movement exceeds the amount of movement threshold, the flag is turned OFF. Includes. [Effects of the Invention]

[0007] According to one embodiment of this disclosure, the technology for determining a person's posture is improved. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing a schematic configuration of a vehicle control device according to one embodiment of the present disclosure. [Figure 2] This flowchart shows the operation of the vehicle control device according to this embodiment. [Figure 3] This flowchart shows the operation of a vehicle control device according to another embodiment. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings.

[0010] (Summary of this embodiment) Referring to Figure 1, an overview of the vehicle control device 1 according to the embodiment of this disclosure will be described. The vehicle control device 1 is an electronic device mounted in a vehicle, such as a computer. The vehicle control device 1 detects an object from an image inside the vehicle using any object detection technology. The image may be a still image or a moving image. The object is the occupant inside the vehicle.

[0011] The vehicle is any vehicle capable of carrying one or more occupants, such as a car, bus, or shuttle bus. The vehicle may be an autonomous vehicle capable of autonomous driving at levels 1 to 5 as defined by the Society of Automotive Engineers (SAE). The vehicle may also be a manually driven vehicle at level 0. The vehicle may be remotely monitored by an observer outside the vehicle. The vehicle may be a vehicle specifically designed for MaaS (Mobility as a Service).

[0012] First, an overview of this embodiment will be described. The vehicle control device 1 according to this embodiment comprises a control unit 10 and an imaging unit 12 that images the interior of the vehicle. The control unit 10 detects an object from the image captured by the imaging unit 12 using object detection technology. The control unit 10 calculates a seating likelihood from the image, indicating the likelihood that the object's posture is a seated posture. Based on whether the seating likelihood is equal to or greater than a likelihood threshold, the control unit 10 decides whether to perform a safety posture determination based on the object's movement amount. Performing a safety posture determination based on the object's movement amount includes calculating the object's movement amount, and if the movement amount is less than or equal to a movement amount threshold, turning on a flag indicating that the object's posture is a safe posture, and if the movement amount exceeds the movement amount threshold, turning off the flag.

[0013] According to this embodiment, in order to determine a safe posture, not only is seating determination based on seating likelihood performed, but safe posture determination is also performed based on the amount of movement of the object. When the amount of movement of the object is small, the likelihood of it affecting the safe operation of the vehicle is low. Therefore, even if the accuracy of seating determination decreases, it is possible to improve the accuracy of safe posture determination by using safe posture determination based on the amount of movement of the object in combination.

[0014] (Configuration of the vehicle control device 1) The vehicle control device 1 comprises a control unit 10, an imaging unit 12, a communication unit 14, and a storage unit 16. Each unit is connected to the others via an in-vehicle network such as CAN (Controller Area Network) or a dedicated line, enabling communication between them.

[0015] The control unit 10 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or a GPU (graphics processing unit), or a dedicated processor specialized for specific processing. The control unit 10 executes processing related to the operation of the vehicle control device 1 while controlling each part of the vehicle control device 1.

[0016] The imaging unit 12 is an arbitrary imaging module installed in the vehicle and capable of imaging some or all of the seats and objects in the vehicle. The imaging module includes one or more cameras. In the present embodiment, the imaging unit 12 is one camera installed on the ceiling of the vehicle. In the present embodiment, the imaging unit 12 captures RGB images. The imaging unit 12 may include a ranging device such as a depth sensor or a stereo camera that acquires depth images.

[0017] The communication unit 14 includes at least one communication interface connected to the in-vehicle network. The communication interface corresponds to, for example, a mobile communication standard such as 4G (4th generation) or 5G (5th generation), a V2X (vehicle-to-everything) communication standard such as DSRC (dedicated short range communications) or cellular V2X, or a wireless LAN (local area network) communication standard such as IEEE8 ()2.11 (Institute of Electrical and Electronics Engineers 802.11).

[0018] The storage unit 16 includes one or more memories. Each memory included in the storage unit 16 may function as, for example, a main memory device, an auxiliary memory device, or a cache memory. The storage unit 16 stores any information used for the operation of the vehicle control device 1. For example, the storage unit 16 stores a system program, an application program, embedded software, and any data used for target detection and pose estimation. The storage unit 16 may store in advance information on the position and shape of each seat. The information stored in the storage unit 16 may be updated with information acquired from an in-vehicle network or an external network via the communication unit 14. In the present embodiment, the state (ON or OFF) of a flag indicating that the posture of the target is a safe posture is updated by the control unit 10 and stored in the storage unit 16. The safe posture includes a sitting posture which is the posture when the target is sitting on the seat, and a stationary posture which is the posture when the target is stationary or when the amount of movement of the target is small.

[0019] In the present embodiment, the storage unit 16 stores in advance an object detection AI for detecting an object and the skeleton of the object included in the captured image, and a pose estimation AI for estimating the pose of the object. The skeleton can be detected from an RGB image. In order to improve the accuracy, a depth image may also be used together for the detection of the skeleton. The object detection AI may include any object detection model such as YOLO (You Only Look Once) or CNN (Convolutional Neural Network). The pose estimation AI may include any pose estimation model for estimating the pose of the object from the skeleton of the object.

[0020] (Operation flow of the vehicle control device 1 according to the present embodiment) [[ID=**10**]]Referring to FIG. 2, the operation of the vehicle control device 1 according to the present embodiment will be described. The control unit 10 executes the following S101 to S109 for each target in the vehicle while the vehicle is temporarily stopped at, for example, a stop, and determines whether the posture of each target is a safe posture. Hereinafter, the communication between the respective units of the vehicle control device 1 is performed via the communication unit 14 and the in-vehicle network.

[0021] S101: The control unit 10 of the vehicle control device 1 acquires the image captured by the imaging unit 12.

[0022] S102: The control unit 10 detects an object from the image using object detection technology.

[0023] The control unit 10 inputs the image captured by the imaging unit 12 to the object detection AI pre-stored in the memory unit 16. The object detection AI detects the presence or absence of an object in the image based on the input image. The object detection AI may further detect the skeleton of the object based on the input image.

[0024] S103: The control unit 10 determines from the image whether the object is facing forward relative to the imaging unit 12. If the object is facing forward, the process proceeds to S106. If the object is not facing forward, the process proceeds to S104.

[0025] "Front direction" refers to a direction inclined relative to the direction from the object towards the camera, for example, within a range of 0 degrees to less than 45 degrees. If the object is not facing the front direction, this includes cases where the object is facing sideways. "Sideways direction" refers to a direction inclined relative to the direction from the object towards the camera, for example, within a range of 45 degrees to less than 135 degrees.

[0026] Due to limitations in the camera's field of view or image processing, capturing characteristic points of a person's posture from the front can be difficult compared to capturing those characteristic points from another direction, such as the side. Therefore, when the subject is facing forward to the imaging unit 12, the accuracy of the calculated seating likelihood may be low. In this embodiment, when the subject is facing forward to the imaging unit 12, seating determination based on the potentially low-accuracy seating likelihood is not performed, and only safety posture determination based on the subject's movement is performed. This reduces the processing load on the vehicle control device 1. Any detection technique such as YOLO or CNN may be used to determine the orientation of the subject. The control unit 10 may, for example, determine whether the subject is facing forward to the imaging unit 12 based on the subject's characteristic quantities in the image.

[0027] S104: The control unit 10 calculates the seating likelihood from the image, which indicates the likelihood that the subject's posture is a seated posture.

[0028] The control unit 10 inputs the image captured by the imaging unit 12 to the posture estimation AI pre-stored in the memory unit 16. The posture estimation AI calculates the likelihood of sitting based on the input image. The control unit 10 may calculate the likelihood of sitting by another method.

[0029] S105: The control unit 10 determines whether the seating likelihood is greater than or equal to the likelihood threshold. If the seating likelihood is greater than or equal to the likelihood threshold (S105-Yes), the process proceeds to S106. If the seating likelihood is less than the likelihood threshold (S105-No), the process proceeds to S109.

[0030] In S105, a seating determination is performed based on the seating likelihood. In this embodiment, if the seating likelihood is greater than or equal to the likelihood threshold, a safety posture determination is performed based on the amount of movement of the subject; if the seating likelihood is less than the likelihood threshold, a safety posture determination based on the amount of movement is not performed. In this embodiment, even when the seating likelihood is greater than or equal to the likelihood threshold, a safety posture determination based on the amount of movement is also performed, thereby increasing the accuracy of the safety posture determination.

[0031] S106: The control unit 10 calculates the amount of movement of the target.

[0032] The control unit 10 may calculate the amount of movement of the target by calculating the amount of movement per unit time for each feature point (e.g., joint) in the target skeleton and summing the amount of movement per unit time for all feature points. When summing the amount of movement for all feature points, different weighting coefficients may be assigned to each feature point. For example, since the target is safer if the amount of movement in the lower body is smaller than that in the upper body, a larger weighting coefficient may be assigned to the joints of the lower body (joints near the waist, knees, feet, etc.) than to the joints of the upper body.

[0033] S107: The control unit 10 determines whether the amount of movement is less than or equal to the amount of movement threshold. If the amount of movement is less than or equal to the amount of movement threshold (S107-YES), the process proceeds to S108. If the amount of movement exceeds the amount of movement threshold (S107-NO), the process proceeds to S109.

[0034] In S107, a safety posture is determined based on the amount of movement of the object. If the amount of movement is below the movement threshold, that is, if the object's posture is stationary, the object can be considered safe. Therefore, a stationary posture can qualify as a safe posture. If the object's posture is safe, the object may or may not be seated.

[0035] S108: The control unit 10 turns on a flag indicating that the target's posture is a safe posture. The process then terminates.

[0036] S109: The control unit 10 turns the flag OFF. The process then returns to S101.

[0037] The control unit 10 repeats steps S101 to S109 until the flag is turned ON.

[0038] The control unit 10 executes processes S101 to S109 for each object in the vehicle. If the flag is ON for all objects, the control unit 10 permits the vehicle to start. If the flag is OFF for at least one object, the control unit 10 does not permit the vehicle to start.

[0039] (Operation flow of vehicle control device 1 according to another embodiment) Referring to Figure 3, the operation of the vehicle control device 1 according to another embodiment will be described. The configuration of the vehicle control device 1 according to the other embodiment is the same as the configuration shown in Figure 1, so the description will be omitted. Among the operations of the vehicle control device 1 according to the other embodiment, the processes S201 to S204 and S206 to S209 are the same as the processes S101 to S104 and S106 to S109, respectively, so the description will be omitted.

[0040] S205: The control unit 10 determines whether the seating likelihood is greater than or equal to the likelihood threshold. If the seating likelihood is greater than or equal to the likelihood threshold (S205-Yes), the process proceeds to S208. If the seating likelihood is less than the likelihood threshold (S205-No), the process proceeds to S206.

[0041] In another embodiment, if the seating likelihood is less than the likelihood threshold, a safety posture determination based on the amount of movement of the object is performed, and if the seating likelihood is equal to or greater than the likelihood threshold, a safety posture determination based on the amount of movement is not performed. In another embodiment, since a safety posture determination based on the amount of movement is not performed if the likelihood is equal to or greater than the likelihood threshold, the processing load on the vehicle control device 1 is reduced.

[0042] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art may make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or separated. For example, in the embodiments described above, an embodiment is also possible in which the configuration and operation of the vehicle control device 1 are distributed among multiple devices or computers that can communicate with each other.

[0043] For example, in the embodiment described above, the control unit 10 of the vehicle control device 1 may start the vehicle by executing automatic driving control of the vehicle when it permits the vehicle to start. The vehicle control device 1 may also be used to provide Mobility as a Service (MaaS), which is a mobility-based service. [Explanation of symbols]

[0044] 1 Vehicle control device, 10 Control unit, 12 Imaging unit, 14 Communication unit, 16 Storage unit

Claims

1. A vehicle control device comprising a control unit and an imaging unit for imaging the interior of a vehicle, The control unit, Using object detection technology, the object is detected from the image captured by the imaging unit. From the aforementioned image, the likelihood of the subject being in a seated position is calculated. Based on whether the seating likelihood is greater than or equal to the likelihood threshold, a decision is made on whether to perform a safety posture determination based on the amount of movement of the subject. Determining a safe posture based on the amount of movement of the aforementioned object is, Calculating the amount of movement of the aforementioned object, and If the amount of movement is less than or equal to the amount of movement threshold, a flag indicating that the target's posture is a safe posture is turned ON; if the amount of movement exceeds the amount of movement threshold, the flag is turned OFF. Vehicle control devices, including

2. In the vehicle control device according to claim 1, determining whether or not to perform a safety posture determination based on the amount of movement of the object is: If the seating likelihood is greater than or equal to the likelihood threshold, a safety posture determination based on the amount of movement is performed; if the seating likelihood is less than the likelihood threshold, the safety posture determination based on the amount of movement is not performed, and the flag is turned OFF. Vehicle control devices, including

3. In the vehicle control device according to claim 1, determining whether or not to perform a safety posture determination based on the amount of movement of the object is: If the seating likelihood is greater than or equal to the likelihood threshold, the flag is turned ON without performing a safety posture determination based on the amount of movement; if the seating likelihood is less than the likelihood threshold, the safety posture determination based on the amount of movement is performed. Vehicle control devices, including

4. A vehicle control device according to claim 1, wherein the control unit further comprises: If the object is facing forward relative to the imaging unit, the determination is made based on the amount of movement without calculating the likelihood of the object being seated. The aforementioned forward direction is a direction that is inclined within a range of 0 degrees or more and less than 45 degrees with respect to the direction from the object toward the imaging unit, in a vehicle control device.

5. A vehicle control device according to any one of claims 1 to 4, wherein the control unit is If the flag is set to ON for all objects within the vehicle, the vehicle is permitted to start. A vehicle control device that does not permit the vehicle to start if the flag is turned OFF for at least one object within the vehicle.

6. A method for providing MaaS (Mobility as a Service) using the vehicle control device described in claim 1.

Images