VEHICLE CONTROL DEVICE, VEHICLE CONTROL SYSTEM, PROGRAM, AND VEHICLE CONTROL METHOD

The vehicle control system uses skeletal coordinate detection and machine learning to ensure users are seated before allowing vehicle operations, addressing the challenge of accurate seating determination in shared transportation.

JP7579731B2Active Publication Date: 2024-11-08KK TOKAI RIKA DENKI SEISAKUSHO
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2021053146
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-26
Publication Date
2024-11-08
Estimated Expiration
2041-03-26

AI Technical Summary

Technical Problem

Existing vehicle control systems struggle to accurately determine whether a user who has entered a vehicle is seated on a seat, particularly in ride-sharing and self-driving shared taxis and buses.

Method used

A vehicle control system that utilizes an imaging device to capture images of the vehicle interior, detects skeletal coordinates of a user using machine learning algorithms, and sets a movable area based on these coordinates through a calibration process, determining seating by ensuring the coordinates remain within a predefined area for a predetermined period.

Benefits of technology

Accurately determines if a user is seated on a seat, preventing unnecessary vehicle operations such as door closure and startup until the user is properly seated, enhancing safety and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579731000001
    Figure 0007579731000001
  • Figure 0007579731000002
    Figure 0007579731000002
  • Figure 0007579731000003
    Figure 0007579731000003
Patent Text Reader

Abstract

To provide a vehicle control device which accurately determines that a user who rode on a vehicle has sat on a seat.SOLUTION: In an authentication ECU 110 constituting a vehicle controller, a skeleton detection unit 112 acquires skeleton coordinates indicative of a position of person's skeleton from an image captured by a camera 80. In a storage device 125, a set area preliminarily set as an area where person's skeleton coordinates acquired by the skeleton detection unit 112 are movable with the person sitting on a seat is stored. A determination unit 114 determines whether a user who rode on the vehicle has sat on the seat or not from the image captured by the camera 80. When user's skeleton coordinates acquired by the skeleton detection unit 112 are within the set area for a prescribed period of time, the determination unit 114 determines that the user has sat on the seat.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a vehicle control device, a vehicle control system, a program used for controlling a vehicle, and a vehicle control method. [Background technology]

[0002] Japanese Patent Laid-Open Publication No. 2011-32782 (Patent Document 1) discloses a keyless entry device capable of improving security. In this keyless entry device, face matching is performed by comparing a face image captured by a camera with a face image registered in advance. Furthermore, when a person's hand touches the door handle, human body communication is performed between an electronic key carried by the person and a human body communication authentication unit of the vehicle via the person, and the electronic key is authenticated. Then, if face matching is confirmed and the electronic key is authenticated, the door is unlocked. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-32782 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, ride sharing and car sharing have become popular, and it is expected that autonomous shared taxis and buses will become more common in the future. In such vehicles, it may be necessary to make it possible for the vehicle to start moving only after the user gets in and sits in the seat.

[0005] Therefore, an object of the present disclosure is to provide a vehicle control device, a vehicle control system, a program, and a vehicle control method that can accurately determine that a user who has entered the vehicle is seated in a seat. [Means for solving the problem]

[0006] A vehicle control device according to the present disclosure includes a storage device and a processing device. The storage device stores a preset area in which skeletal coordinates indicating the position of a person's skeleton, which are acquired from an image captured of the person seated in a vehicle seat, can move. The processing device executes a seating determination process for determining whether a user who has boarded the vehicle is seated in the seat from an image captured of the interior of the vehicle. The seating determination process includes a process of acquiring the skeletal coordinates of the user from the image captured of the interior of the vehicle, and determining that the user is seated in the seat if the acquired skeletal coordinates of the user are included in the set area for a predetermined period of time.

[0007] The vehicle control system according to the present disclosure includes an imaging device, a storage device, and a processing device. The imaging device captures an image of the interior of the vehicle. The storage device stores a preset area as an area in which skeletal coordinates indicating the position of the skeleton of a person seated in a vehicle seat, which is acquired from an image captured by the imaging device, can move. The processing device includes a processing device that executes a seating determination process that determines whether or not a user who has boarded the vehicle is seated in the seat from the image captured by the imaging device. The seating determination process includes a process of acquiring the skeletal coordinates of the user from the image captured by the imaging device, and determining that the user is seated in the seat when the acquired skeletal coordinates of the user are included in the set area for a predetermined period of time.

[0008] In addition, the program of the present disclosure is a program for causing a computer to execute the steps of: acquiring a set area that is set as an area in which skeletal coordinates indicating the position of the skeletal structure of a person seated in a vehicle seat can move, which is acquired from an image captured of the person; acquiring skeletal coordinates of a user riding in the vehicle from an image captured of the interior of the vehicle cabin; and determining that the user is seated in the seat if the acquired skeletal coordinates of the user are included within the set area for a predetermined period of time.

[0009] In addition, the vehicle control method of the present disclosure includes a step of acquiring a set area in which skeletal coordinates indicating the position of the skeletal structure of a person seated in a vehicle seat can move, the set area being acquired from an image captured of the person, acquiring skeletal coordinates of a user riding in the vehicle from an image captured of the interior of the vehicle cabin, and determining that the user is seated in the seat if the acquired skeletal coordinates of the user are included within the set area for a predetermined period of time. Effect of the Invention

[0010] According to the vehicle control device, vehicle control system, program, and vehicle control method disclosed herein, it is possible to accurately determine whether a user who has entered the vehicle has sat in the seat, based on a preset set area as an area in which the skeletal coordinates of a person seated in the seat may move. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing an example of an overall configuration of a vehicle to which a control device according to an embodiment of the present disclosure is applied; [Diagram 2] FIG. 2 is a block diagram functionally illustrating a configuration of an authentication ECU. [Diagram 3] FIG. 11 is a diagram showing the skeleton of a person seated in a seat being detected; [Figure 4] 11 is a diagram showing an example of data set by a calibration processing unit and stored in a storage device; FIG. [Diagram 5] 10 is a flowchart showing an example of a procedure of a calibration process executed by an authentication ECU. [Figure 6] 6 is a flowchart showing an example of a procedure of a seating determination process executed by an authentication ECU. [Figure 7] 10 is a flowchart illustrating a condition for executing a predetermined process scheduled after a user gets on the train. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and their description will not be repeated.

[0013] In the following, an example will be described in which a small shared vehicle operated by autonomous driving is used as a vehicle (mobile service) to which a control device according to an embodiment of the present disclosure is applied.

[0014] FIG. 1 is a diagram showing an example of an overall configuration of a vehicle to which a control system according to an embodiment of the present disclosure is applied. Referring to FIG. 1, vehicle 1 is an electric small passenger vehicle operated by automatic driving. Vehicle 1 includes a motor generator (hereinafter referred to as "MG (Motor Generator)") 10, a power transmission gear 20, a drive shaft 30, drive wheels 40, a power control unit (hereinafter referred to as "PCU (Power Control Unit)") 50, and a battery 60. Vehicle 1 further includes a camera 80, a control device 100, an HMI (Human Machine Interface) device 200, and a GPS (Global Positioning System) device 300.

[0015] The MG 10 is, for example, a synchronous motor with a permanent magnet embedded in the rotor, and has a function as an electric motor (motor) and a function as a generator. The output torque of the MG 10 is transmitted to the drive wheels 40 via a power transmission gear 20 including a reduction gear, a differential device, etc., and a drive shaft 30.

[0016] When braking the vehicle 1, the MG 10 is driven by the drive wheels 40 and operates as a generator. This allows the MG 10 to function as a braking device that performs regenerative braking to convert the kinetic energy of the vehicle 1 into electric power. The regenerative electric power generated by the regenerative braking force in the MG 10 is stored in the battery 60.

[0017] The PCU 50 is a power conversion device that converts power bidirectionally between the MG 10 and the battery 60. The PCU 50 includes, for example, an inverter and a converter (neither of which are shown) that operate based on a control signal from an MGECU (Electronic Control Unit) 140 (described later) included in the control device 100.

[0018] The converter boosts the voltage received from the battery 60 and outputs the boosted voltage to the inverter when the battery 60 is discharging. The inverter converts the DC power supplied from the converter into AC power to drive the MG 10.

[0019] On the other hand, the inverter converts AC power generated by MG 10 into DC power and supplies it to the converter when charging battery 60. The converter steps down the power received from the inverter to a voltage suitable for charging battery 60 and supplies it to battery 60.

[0020] Furthermore, the PCU 50 suspends charging and discharging by stopping the operation of the inverter and the converter based on a control signal from the MGECU 140. Note that the PCU 50 may be configured without the converter.

[0021] The battery 60 stores power for driving the MG 10. The battery 60 is a rechargeable DC power source (secondary battery) and is configured by stacking a plurality of single cells (battery cells) and electrically connecting them in series, for example. The battery 60 is configured to be chargeable by a charging device (not shown) using power supplied from a power source external to the vehicle.

[0022] The vehicle 1 is provided with an entry / exit door 3 on one of the left and right sides of the vehicle body 2. The entry / exit door 3 is opened and closed by a door actuator 90. A camera 80 that captures an image of the interior of the vehicle compartment 4 is provided in the vehicle compartment 4. The camera 80 captures an image of a person (subject) riding in the vehicle 1 to obtain image data. The camera 80 is, for example, a digital camera that includes an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an optical system (lens, aperture, shutter, etc.) that forms an image of the subject on the imaging element.

[0023] The HMI device 200 is a device that provides the user with various types of information related to the vehicle 1. The HMI device 200 is typically a display provided in the vehicle interior 4, and also includes a speaker and the like. The HMI device 200 also functions as a touch panel and microphone that can be operated by the user, and can transmit, for example, the requests of the occupants to the vehicle 1 or a server 400 outside the vehicle by the user's touch panel operation or voice.

[0024] The GPS device 200 calculates the current position (own vehicle position) of the vehicle 1 based on the GPS information. The vehicle 1 is configured to travel along a travel route by automatic driving using the current position and surrounding information of the vehicle 1. The surrounding information of the vehicle 1 is acquired, for example, from a LiDAR (Light Detection and Ranging) device, a white line recognition camera, a millimeter wave radar, etc. (none of which are shown).

[0025] The control device 100 includes an authentication ECU 110, a communication device 120, a storage device 125, a door ECU 130, and an MGECU 140. The authentication ECU 110 includes a CPU (Central Processing Unit) and memories (ROM (Read Only Memory) and RAM (Random Access Memory)). The CPU loads a program stored in the ROM into the RAM or the like and executes the program. Various processes executed by the authentication ECU 110 are described in the program stored in the ROM.

[0026] The authentication ECU 110 performs face authentication of a user who gets into the vehicle 1, using an image captured by the camera 80. Then, after face authentication of the user is performed, the authentication ECU 110 further uses the image captured by the camera 80 to determine whether the user is seated in a seat.

[0027] Specifically, the authentication ECU 110 acquires skeletal coordinates indicating the positions of the bones (shoulders, neck, face, etc.) of the user from the image captured by the camera 80. Then, when the acquired skeletal coordinates of each part (shoulders, neck, face, etc.) are included within a predetermined set area set for each seat for a predetermined period of time, the authentication ECU 110 determines that the user is seated in the seat.

[0028] The above-mentioned set area is set for each seat by a calibration process described later, and is stored in the storage device 125. This set area is set in advance as an area in which the skeleton coordinates of a person, acquired from an image captured by the camera 80, can move when the person is seated in the seat. The storage device 125 stores the set area thus set in advance for each seat. The storage device 125 is, for example, a hard disk drive (HDD) or a solid state drive (SSD). The configuration of the authentication ECU 110 will be described in detail later.

[0029] The communication device 120 includes various communication I / Fs (interfaces). The communication device 120 may include a DCM (Data Communication Module) or a communication I / F compatible with 5G (fifth generation mobile communication system). The authentication ECU 110 can wirelessly communicate with a server 400 outside the vehicle through the communication device 120.

[0030] The door ECU 130 also includes a CPU and memories (ROM and RAM) like the authentication ECU 110. The door ECU 130 receives a command from the authentication ECU 110 to drive the door actuator 90 and control the opening and closing of the getting-into / out door 3.

[0031] The MGECU 140 also includes a CPU and memory (ROM and RAM) like the authentication ECU 110. The MGECU 140 receives signals and commands from the authentication ECU 110, the GPS device 300, etc., and controls the operation of the PCU 50 based on information such as maps and programs stored in the memory.

[0032] The server 400 can communicate with the vehicle 1. The server 400 can also communicate with the user's mobile terminal 500. A user of the vehicle 1, which is a small shared vehicle, can make a reservation for using the vehicle 1 from the mobile terminal 500. For example, when a user starts an application on the mobile terminal 500, inputs a boarding location, boarding date and time, a disembarking location, and the like, and taps a reservation button, the reservation information is transmitted from the mobile terminal 500 to the server 400. When the server 400 receives the reservation information from the mobile terminal 500, it completes the reservation for the user.

[0033] 2 is a block diagram functionally showing a configuration of authentication ECU 110. With reference to FIG. 2, authentication ECU 110 includes a face authentication unit 111, a skeleton detection unit 112, a calibration processing unit 113, and a determination unit 114.

[0034] The face authentication unit 111 detects a face image from an image captured by the camera 80 and compares it with a face image of the user acquired from the server 400 (FIG. 1) via the communication device 120, thereby authenticating the face of the user who has boarded the vehicle 1. As a method for detecting a face image from an image captured by the camera 80, for example, the face image can be detected using a machine learning algorithm such as deep learning using a convolutional neural network (CNN). The face image of the user acquired from the server 400 is registered in the server 400 when the user registers for use or makes a reservation for use.

[0035] The skeleton detection unit 112 detects the skeleton of the person sitting in the seat from the image captured by the camera 80. Skeleton detection can be performed using a known skeleton detection algorithm that uses machine learning such as deep learning. By skeleton detection, the position (coordinates) of each part (shoulders, neck, face, etc.) of the person sitting in the seat can be obtained. Hereinafter, the position (coordinates) of each part of the person's skeleton obtained by skeleton detection will be referred to as "skeleton coordinates."

[0036] The calibration processing unit 113 executes a process (calibration process) for setting an area in which the skeletal coordinates of a person can move when the person is seated in the seat. The area (hereinafter referred to as the "set area") set by the calibration processing unit 113 is used for seating determination, which will be described later, that is executed after a user who has made a reservation to use the vehicle 1 gets into the vehicle 1 and face authentication is performed. The calibration process is performed in advance before the vehicle 1 starts operating, and information on the set area is stored in the storage device 125.

[0037] When a person is seated in the seat, the calibration processing unit 113 acquires from the skeleton detection unit 112 the skeleton coordinates detected by the skeleton detection unit 112 from an image captured by the camera 80. The calibration processing unit 113 acquires the skeleton coordinates from the skeleton detection unit 112 for a certain period of time, and sets an area in which the skeleton coordinates of the person may move in the seat in which the person is seated, based on an area in which the skeleton coordinates have moved during that period.

[0038] Such an area is set for each part (shoulder, neck, face, etc.) of the person sitting in the seat. Taking the face as an example, the facial skeletal coordinates acquired over a certain period of time fluctuate, and a rectangular area is set based on the maximum and minimum coordinate values ​​during that period. The calibration processing unit 113 sets such an area in which the skeletal coordinates can move for each seat, and stores the set area (for each part) for each seat in the storage device 125.

[0039] The determination unit 114 determines whether or not a user of the vehicle 1 has completed sitting in the seat after the user gets into the vehicle 1 and face authentication is performed by the face authentication unit 111. Specifically, the face authentication unit 111 performs face authentication of the user who gets into the vehicle 1 from an image captured by the camera 80, and when the skeleton detection unit 112 detects the skeleton of the user who has passed the face authentication, the determination unit 114 acquires the skeleton coordinates of each part of the user from the skeleton detection unit 112. In addition, the determination unit 114 acquires from the storage device 125 information on the above-mentioned setting area that is preset by the calibration processing unit 113 for the seat in which the user is seated.

[0040] Then, the determination unit 114 determines that the user has completed sitting on the seat when the skeletal coordinates of the user acquired from the skeleton detection unit 112 are included for a predetermined period within a set area acquired from the storage device 125. Whether or not the user's skeletal coordinates are included for a predetermined period within a set area is determined for each part of the user, and when the skeletal coordinates of each part (e.g., shoulders, neck, face) are included within the corresponding set area for a predetermined period, it is determined that the user has completed sitting on the seat. The determination unit 114 performs such sitting determination for each seat.

[0041] When the judgment unit 114 determines that all users in the vehicle 1 have completed taking their seats, it outputs a command to the door ECU 130, the MGECU 140, etc. to execute the processing scheduled after the users have boarded the vehicle 1 (such as closing the boarding / alighting door 3 (Figure 1) and starting the vehicle).

[0042] When the door ECU 130 receives the execution command from the determination unit 114, it drives the door actuator 90 to close the getting-into / out door 3. When the MGECU 140 receives the execution command from the determination unit 114, it controls the PCU 50 (FIG. 1) to start traveling of the vehicle 1, on the condition that the getting-into / out door 3 is closed.

[0043] Fig. 3 is a diagram showing a state in which the skeleton of a person sitting on a seat is detected. Referring to Fig. 3, as described above, the skeleton of the person sitting on the seat is detected by skeleton detection unit 112, and the detection result is displayed, for example, on display screen 210 of HMI device 200, superimposed on the captured image of camera 80.

[0044] A position on the display image of the display screen 210 is indicated, for example, by XY coordinates. In this example, the X axis and the Y axis are set along the horizontal and vertical directions of the display image, respectively, and the origin is set at an appropriate position (for example, the lower left of the display image).

[0045] In the displayed image, point 212 indicates the skeletal coordinates of the right shoulder detected by skeleton detection, point 214 indicates the skeletal coordinates of the left shoulder detected by skeleton detection, point 216 indicates the skeletal coordinates of the neck detected by skeleton detection, and point 218 indicates the skeletal coordinates of the face detected by skeleton detection.

[0046] Area 222 is an area set by calibration processing unit 113 as an area in which the skeletal coordinates of the right shoulder of a person seated in a seat can move. In this example, the skeletal coordinates of the right shoulder are acquired for a certain period of time, and rectangular area 222 is set from the maximum and minimum values ​​of the X coordinate and the maximum and minimum values ​​of the Y coordinate during that period.

[0047] Area 224 is an area set by calibration processing unit 113 as an area in which the skeletal coordinates of the left shoulder of a person sitting in a seat can move. As in the case of the right shoulder, the skeletal coordinates of the left shoulder are acquired for a certain period of time, and rectangular area 224 is set from the maximum and minimum values ​​of the X coordinate and the maximum and minimum values ​​of the Y coordinate during that period.

[0048] Area 226 is an area set by calibration processing unit 113 as an area in which the skeletal coordinates of the neck of a person seated in a seat can move. As in the case of the right shoulder, the skeletal coordinates of the neck are acquired for a certain period of time, and rectangular area 226 is set from the maximum and minimum values ​​of the X coordinate and the maximum and minimum values ​​of the Y coordinate during that period.

[0049] Area 228 is an area set by calibration processing unit 113 as an area in which the skeletal coordinates of the face of a person sitting in the seat can move. As in the case of the right shoulder, the skeletal coordinates of the face are acquired for a certain period of time, and rectangular area 228 is set from the maximum and minimum values ​​of the X coordinate and the maximum and minimum values ​​of the Y coordinate during that period.

[0050] When the skeletal coordinates 212-218 of each part (right shoulder, left shoulder, neck, face) of a user riding in vehicle 1 are included within areas 222-228, respectively, for a predetermined period of time, the judgment unit 114 (Figure 2) judges that the user has completed sitting in the seat.

[0051] Fig. 4 is a diagram showing an example of data set by calibration processing unit 113 and stored in storage device 125. Referring to Fig. 4, this example shows data on areas set by calibration processing unit 113 for each part on a first sheet (sheet 1) out of multiple sheets.

[0052] Specifically, the calibration processing unit 113 acquires the skeletal coordinates of the right shoulder of the person seated on the first seat for a certain period of time, and stores the maximum value X1 and minimum value X2 of the X coordinate and the maximum value Y1 and minimum value Y2 of the Y coordinate during that period. The area 222 shown in FIG. 3 is set from these data X1, X2, Y1, and Y2.

[0053] The same applies to the data for the left shoulder, neck, and face. The same data as the first sheet is stored for the other sheets.

[0054] 5 is a flowchart showing an example of the procedure of the calibration process executed by the authentication ECU 110. A series of processes shown in this flowchart is written in a program and stored in the memory of the authentication ECU 110. The process of this flowchart is performed in advance, for example, during development or delivery of the vehicle 1, prior to operation of the vehicle 1. In addition, the process of this flowchart is performed for each seat of the vehicle 1.

[0055] 5, the authentication ECU 110 acquires an image of a person seated in a seat from the camera 80 (step S10). Next, the authentication ECU 110 executes a skeleton detection process for detecting the skeleton of the person seated in the seat from the acquired image (step S15). As described above, the skeleton detection can be performed using a known skeleton detection algorithm that uses machine learning such as deep learning. Note that in this calibration process, face authentication by the face authentication unit 111 shown in FIG. 2 is not performed.

[0056] The authentication ECU 110 executes a skeleton detection process to acquire skeleton coordinates of each part (shoulder, neck, face, etc.) of the person seated in the seat (step S20). Next, the authentication ECU 110 determines whether the acquired skeleton coordinates are greater than the maximum value of the skeleton coordinates stored in the storage device 125 (step S25).

[0057] The processes from step S25 to step S50 described later are performed for each body part acquired in step S20. Hereinafter, in the processes from step S25 to step S50, "skeletal coordinates" refers to the skeletal coordinates of the target body part. That is, to explain the process of step S25 in detail, the authentication ECU 110 determines whether the skeletal coordinates (X, Y) of the target body part (e.g., right shoulder) acquired in step S20 are greater than the maximum value (X1, Y1 in FIG. 2) of the skeletal coordinates of the target body part (right shoulder) stored in the storage device 125.

[0058] If it is determined in step S25 that the acquired skeletal coordinates are greater than the maximum value of the skeletal coordinates stored in the storage device 125 (YES in step S25), the authentication ECU 110 updates (saves) the maximum value of the skeletal coordinates stored in the storage device 125 with the acquired skeletal coordinates (step S30). For example, if the value of the skeletal coordinate X of the acquired target body part (e.g., the right shoulder) is greater than the maximum value X1, the maximum value X1 is updated with the value of the skeletal coordinate X. A similar process is performed for the skeletal coordinate Y.

[0059] When the acquired skeleton coordinates are equal to or less than the maximum value of the skeleton coordinates stored in the storage device 125 (NO in step S25), the authentication ECU 110 proceeds to step S35 without executing step S30. When the maximum value of the skeleton coordinates is not stored in the storage device 125, the processes of steps S25 and S30 are not executed, and the skeleton coordinates acquired in step S20 are stored in the storage device 125 as a temporary maximum value.

[0060] Next, the authentication ECU 110 determines whether or not the skeletal coordinates acquired in step S20 are smaller than the minimum value of the skeletal coordinates stored in the storage device 125 (step S35). More specifically, the authentication ECU 110 determines whether or not the skeletal coordinates (X, Y) of the target body part (e.g., right shoulder) acquired in step S20 are smaller than the minimum value (X2, Y2 in FIG. 2) of the skeletal coordinates of the target body part (right shoulder) stored in the storage device 125.

[0061] If it is determined in step S35 that the acquired skeletal coordinates are smaller than the minimum value of the skeletal coordinates stored in the storage device 125 (YES in step S35), the authentication ECU 110 updates (saves) the minimum value of the skeletal coordinates stored in the storage device 125 with the acquired skeletal coordinates (step S40). For example, if the value of the acquired skeletal coordinate X of the target body part (e.g., the right shoulder) is smaller than the minimum value X2, the minimum value X2 is updated with the value of the skeletal coordinate X. A similar process is performed for the skeletal coordinate Y.

[0062] When the acquired skeleton coordinates are equal to or greater than the minimum value of the skeleton coordinates stored in the storage device 125 (NO in step S35), the authentication ECU 110 proceeds to step S45 without executing step S40. When the minimum value of the skeleton coordinates is not stored in the storage device 125, the processes of steps S35 and S40 are not executed, and the skeleton coordinates acquired in step S20 are stored in the storage device 125 as a provisional minimum value.

[0063] Next, the authentication ECU 110 determines whether or not to acquire the next image from the camera 80 (step S45). If it is determined that the next image is to be acquired (YES in step S45), the authentication ECU 110 returns the process to step S10 and acquires the next image from the camera 80. Note that whether or not to acquire the next image from the camera 80 and execute the processes from step S10 onwards may be determined based on a time period from the start of the series of processes, or based on the number of images (number of frames) acquired from the camera 80.

[0064] In step S45, when it is determined that it is not necessary to obtain the next image from the camera 80 (NO in step S45), the authentication ECU 110 creates a movable area for each body part from the skeletal coordinates (FIG. 2) stored in the storage device 125 (step S50). Specifically, referring again to FIG. 2, for example, for the skeletal coordinates of the right shoulder, a rectangular area determined by the maximum value X1 and minimum value X2 of the X coordinate and the maximum value Y1 and minimum value Y2 of the Y coordinate is created as the movable area of ​​the right shoulder.

[0065] Then, the authentication ECU 110 sets the movable area created for each part for the target seat (step S55). Note that the setting of the movable area for each part is stored in the storage device 125 for each seat.

[0066] 6 is a flowchart showing an example of the procedure of the seating determination process executed by the authentication ECU 110. The series of processes shown in this flowchart are also written in a program and stored in the memory of the authentication ECU 110. The process of this flowchart is performed for each seat in which a user (passenger) whose face has been authenticated by the face authentication unit 111 is detected.

[0067] 6, the authentication ECU 110 acquires the movable area for each part of the target seat, which is set in the calibration process (step S55 in FIG. 5), from the storage device 125 (step S110).

[0068] Next, the authentication ECU 110 acquires an image of the user seated in the seat from the camera 80 (step S115). Then, the authentication ECU 110 executes a skeleton detection process to detect the skeleton of the user from the acquired image (step S120). Here, similar to the skeleton detection process executed in the calibration process (step S15 in FIG. 5), the skeleton detection can be performed using a known skeleton detection algorithm that uses machine learning such as deep learning.

[0069] Next, the authentication ECU 110 executes a skeleton detection process to acquire skeleton coordinates of each part (shoulder, neck, face, etc.) of the user seated on the seat (step S125).Then, the authentication ECU 110 determines whether the acquired skeleton coordinates are within the range of the movable area acquired in step S110 (step S130).

[0070] The processes of step S130 and step S135 described below are performed for each body part acquired in step S125. That is, to explain the process of step S130 in detail, authentication ECU 110 determines whether or not the skeletal coordinates (X, Y) of the target body part (e.g., right shoulder) acquired in step S125 are within the movable area of ​​the target body part (right shoulder) acquired in step S110. If the acquired skeletal coordinates are not within the movable area (NO in step S130), authentication ECU 110 returns the process to step S115.

[0071] If it is determined in step S130 that the acquired skeleton coordinates are within the movable area (YES in step S130), authentication ECU 110 determines whether or not a predetermined period of time has elapsed during which the acquired skeleton coordinates are within the movable area (step S135). The predetermined period of time may be determined by the time from the start of the series of processes, or by the number of images (number of frames) acquired in step S115. If the predetermined period of time has not elapsed (NO in step S135), authentication ECU 110 returns the process to step S115.

[0072] When it is determined in step S135 that the predetermined period has elapsed (YES in step S135), authentication ECU 110 determines whether or not the predetermined period has elapsed for all parts acquired in step S125 (step S140). When there is a part for which the predetermined period has not elapsed (NO in step S140), authentication ECU 110 returns the process to step S115.

[0073] If it is determined in step S140 that the above-mentioned predetermined period has elapsed for all parts (YES in step S140), authentication ECU 110 determines that sitting in the seat has been completed (step S145).

[0074] Then, when it is determined that all the passengers who boarded the vehicle 1 have completed taking their seats, a predetermined process scheduled after the passengers board the vehicle 1 is executed.

[0075] 7 is a flowchart for explaining the execution conditions of a predetermined process scheduled after a user gets in. The series of processes shown in this flowchart are also written in a program and stored in the memory of the authentication ECU 110. The process of this flowchart is executed when the seating determination process shown in FIG.

[0076] 7, the authentication ECU 110 judges whether or not the user (passenger) whose face has been authenticated by the face authentication unit 111 has completed sitting in all the seats where the user has been detected (step S210). If it is judged that the user has not completed sitting in all the seats (NO in step S210), the authentication ECU 110 judges whether or not a predetermined time has elapsed (step S230). This predetermined time is appropriately determined based on the average time until the user is determined to be seated by the seating determination process.

[0077] If the predetermined time has not elapsed (NO in step S230), the process returns to step S210. On the other hand, if it is determined in step S230 that the predetermined time has elapsed (YES in step S230), the authentication ECU 110 outputs a command for alarm processing to notify the user who has boarded the vehicle 1 that he has not yet sat in the seat (step S240). For example, when the HMIECU (not shown) receives such a command for alarm processing, it outputs a notification to encourage the user who has boarded the vehicle 1 to sit in the seat using a display, speaker, etc. of the HMI device 200 (FIG. 1).

[0078] On the other hand, in step S210, when it is determined that the user (passenger) whose face has been authenticated by the face authentication unit 111 has completed sitting in all detected seats (YES in step S210), the authentication ECU 110 outputs an execution command for a predetermined process scheduled after the user gets into the vehicle 1 (step S220). For example, the door ECU 130 (FIG. 1) that has received the execution command closes the boarding / alighting door 3, and the MGECU 140 (FIG. 1) that has received the execution command drives the MG 10 to start the vehicle 1, on the condition that the boarding / alighting door 3 is closed.

[0079] As described above, in the calibration process of this embodiment, the skeleton detection unit 112 detects the skeleton coordinates of a person seated in a seat of the vehicle 1 from an image captured by the camera 80 of the person. Then, a setting area is obtained as an area within which the detected skeleton coordinates can move, and is stored in the storage device 125.

[0080] When a user gets into the vehicle 1, a seating determination process is executed to determine whether or not the user is seated in a seat from an image captured by the camera 80. In the seating determination process, the skeleton detection unit 112 acquires the skeleton coordinates of the user from the image captured by the camera 80, and if the acquired skeleton coordinates of the user are included within a set area acquired by the calibration process for a predetermined period of time, it is determined that the user is seated in the seat.

[0081] Therefore, according to this embodiment, it is possible to accurately determine whether a user who has boarded vehicle 1 has sat in the seat, based on a set area that is pre-set as an area in which the skeletal coordinates of a person sitting in the seat can move.

[0082] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The technical scope of the present disclosure is defined by the claims, not by the description of the embodiments described above, and is intended to include all modifications within the meaning and scope of the claims.

[0083] The exemplary embodiments described above are examples of the following aspects. A vehicle control system according to one aspect includes an imaging device, a storage device, and a processing device. The imaging device captures an image of the interior of a vehicle cabin. The storage device stores a preset area as an area in which a skeletal coordinate of a person seated in a seat may move, the skeletal coordinate of the person being acquired from an image captured by the imaging device. The processing device includes a processing device that executes a seating determination process that determines whether a user who has boarded the vehicle is seated in the seat from the image captured by the imaging device. The seating determination process includes a process of acquiring the skeletal coordinate of the user from the image captured by the imaging device, and determining that the user is seated in the seat when the acquired skeletal coordinate of the user is included within the set area for a predetermined period of time.

[0084] According to the above control system, it is possible to accurately determine whether a passenger has sat in the seat based on a preset area that is set as an area in which the skeletal coordinates of a person seated in the seat can move.

[0085] The set area may be set for each predetermined body part of a person as an area in which the skeletal coordinates of each body part can move. The seating determination process may include a process of acquiring the skeletal coordinates of each body part of the user from an image captured inside the vehicle cabin, and determining that the user is seated in the seat when the acquired skeletal coordinates of each body part of the user are included in the set area for a predetermined period of time.

[0086] This makes it possible to more accurately determine whether a passenger is seated in the seat based on a preset area that is set as the area in which the skeletal coordinates of each part of the person seated in the seat (shoulders, neck, face, etc.) can move.

[0087] The above-mentioned set area may be set based on an area in which the skeleton coordinates of a person have moved over a certain period of time.

[0088] This makes it possible to avoid the setting area being unnecessarily enlarged or small. When it is determined that the user is seated in the seat by the seating determination process, the processing device may execute a process that is scheduled after the user gets into the vehicle.

[0089] This makes it possible to avoid a process that is scheduled to be performed after the user gets in the vehicle being executed before the user sits down in the seat.

[0090] The scheduled operations may include operations for automatically closing the vehicle doors. This makes it possible to prevent the vehicle door from being closed before the user sits in the seat, and as a result, it is possible to prevent the vehicle from starting to move in conjunction with the door being closed before the user sits in the seat, for example. [Explanation of symbols]

[0091] 1 vehicle, 2 vehicle body, 3 boarding door, 4 passenger compartment, 10 MG, 20 power transmission gear, 30 drive shaft, 40 drive wheel, 50 PCU, 60 battery, 80 camera, 90 door actuator, 100 control device, 110 authentication ECU, 111 face authentication unit, 112 skeleton detection unit, 113 calibration processing unit, 114 judgment unit, 120 communication device, 125 storage device, 130 door ECU, 140 MGECU, 200 HMI device, 210 display screen, 300 GPS device, 400 server, 500 mobile terminal.

Claims

1. a storage device that stores a preset area as an area in which skeletal coordinates indicating a position of a skeleton of a person seated in a vehicle seat, the skeletal coordinates being acquired from an image of the person seated in a vehicle seat, can move; a processing device that executes a calibration process for setting the set area and a seating determination process for determining whether or not a user who has boarded the vehicle is seated in a seat based on an image captured inside the vehicle cabin, the calibration process includes acquiring skeletal coordinates of the person for a certain period of time from an image of the person, and setting an area in which the skeletal coordinates of the person can move as the set area based on an area in which the acquired skeletal coordinates have moved during the certain period of time; The seating determination process includes: Acquire skeletal coordinates of the user from an image captured inside the vehicle cabin; A vehicle control device that includes a process of determining that the user is seated in the seat when the acquired skeletal coordinates of the user are included within the set area for a predetermined period of time.

2. The calibration process sets, for each predetermined part of the person, an area in which a skeletal coordinate of each part can move as the setting area; The seating determination process includes: acquiring skeletal coordinates of each of the body parts of the user from an image captured inside the vehicle interior; 2. The vehicle control device according to claim 1, further comprising a process of determining that the user is seated in the seat when the acquired skeletal coordinates of each part of the user are included within the set area for a predetermined period of time.

3. A vehicle control device as described in claim 1 or claim 2, wherein the calibration process sets the setting area for each seat of the vehicle.

4. 4. A vehicle control device as claimed in claim 1, wherein the processing device executes processing scheduled after the user gets into the vehicle when the seating determination processing determines that the user is seated in the seat.

5. The vehicle control device according to claim 4 , wherein the scheduled processing includes processing for automatically closing a door of the vehicle.

6. An imaging device that captures an image of the interior of a vehicle; a storage device that stores a preset setting area as an area in which a skeleton coordinate indicating a position of a skeleton of a person seated in a seat of the vehicle can move, the skeleton coordinate being acquired from an image captured by the imaging device of the person; a processing device that executes a calibration process for setting the set area and a seating determination process for determining whether or not a user who has boarded the vehicle is seated in a seat based on an image captured by the imaging device, the calibration process includes acquiring skeletal coordinates of the person for a certain period of time from an image of the person, and setting an area in which the skeletal coordinates of the person can move as the set area based on an area in which the acquired skeletal coordinates have moved during the certain period of time; The seating determination process includes: acquiring skeletal coordinates of the user from an image captured by the imaging device; A vehicle control system including a process of determining that the user is seated in a seat when the acquired skeletal coordinates of the user are included within the set area for a predetermined period of time.

7. acquiring skeletal coordinates indicating a position of a skeleton of a person seated in a vehicle seat for a certain period of time from an image of the person, and setting an area in which the skeletal coordinates of the person can move based on an area in which the acquired skeletal coordinates have moved during the certain period of time; acquiring the set area set by the setting step; acquiring skeletal coordinates of a user who is in the vehicle from an image captured inside the vehicle cabin; and a step of determining that the user is seated in the seat if the acquired skeletal coordinates of the user are included within the set area for a predetermined period of time.

8. The setting step includes a step of setting, for each predetermined part of the person, an area in which skeletal coordinates of each part can move as the setting area; The step of acquiring skeletal coordinates includes a step of acquiring skeletal coordinates of each of the body parts of the user from an image captured inside the vehicle cabin, 8. The program according to claim 7, wherein the determining step includes a step of determining that the user is seated in the seat when the acquired skeletal coordinates of each part of the user are included within the set area for a predetermined period of time.

9. A program as described in claim 7 or claim 8, wherein the setting step includes a step of setting the setting area for each seat of the vehicle.

10. 10. The program according to claim 7, further comprising a step of executing a process scheduled after the user gets into the vehicle when it is determined in the determining step that the user is seated in the seat.

11. The computer-readable medium according to claim 10 , wherein the scheduled operations include operations for automatically closing a door of the vehicle.

12. acquiring skeletal coordinates indicating a position of a skeleton of a person seated in a vehicle seat for a certain period of time from an image of the person, and setting an area in which the skeletal coordinates of the person can move based on an area in which the acquired skeletal coordinates have moved during the certain period of time; acquiring the set area set by the setting step; acquiring skeletal coordinates of a user who is in the vehicle from an image captured inside the vehicle cabin; and determining that the user is seated in the seat when the acquired skeletal coordinates of the user are included within the set area for a predetermined period of time.

Citation Information

Patent Citations

  • Keyless entry device

    JP2011032782A

  • Device for estimation of occupant posture

    JP2011123733A

  • Vehicle door opening / closing control device

    JP2020007878A

  • Work process management system, work process management method, and work process management program

    JP2020013341A

  • 3D feature point information generator

    JP2020047273A