Control system and control method of a control system

Abstract

This invention provides a control system and a control method for the control system. In the control of vehicle doors, it is possible to improve vehicle safety while ensuring the convenience of vehicle users. The control system (1) includes: an exterior image acquisition unit (43) that acquires an exterior image; a first detection unit (204) that detects an external person (VP) based on the exterior image; a second detection unit (206) that detects a predetermined action of the detected external person; a first authentication unit (207) that authenticates whether the detected external person is a registered user of the vehicle (V) based on a predetermined part of the detected external person; and a door control unit that performs door opening control of the vehicle, wherein if the detected predetermined action is the predetermined action of the external person authenticated as a registered user, the door control unit performs the door opening control in a first manner based on the predetermined action.

Description

Technical Field

[0001] This invention relates to control systems and control methods for control systems. Background Technology

[0002] In recent years, efforts have intensified to provide sustainable transportation systems that take into account vulnerable groups such as the elderly and children among transportation participants. To achieve this, research and development are focused on further improving the safety and convenience of transportation through advancements related to vehicle boarding and alighting. Furthermore, technologies related to vehicle door control have long been known as techniques for improving convenience (see, for example, Patent Document 1). Patent Document 1 discloses a technology that non-contactly detects the movements of a human body within a defined area of ​​the vehicle, determines the door's movement based on the detected movements, and controls the door drive unit accordingly.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2013-28903 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] According to Patent Document 1, the vehicle user can open the vehicle door with a gesture, thus improving the convenience of the vehicle user. However, Patent Document 1 has the problem that a third party can also open the vehicle door with a gesture.

[0008] The present invention was made in view of the above circumstances, and its object is to improve vehicle safety while ensuring the convenience of vehicle users in the control of vehicle doors.

[0009] Methods for solving problems

[0010] One aspect of the present invention is a vehicle control system comprising: an exterior image acquisition unit that acquires an exterior image of the vehicle; a first detection unit that detects an external person present outside the vehicle based on the exterior image acquired by the exterior image acquisition unit; a second detection unit that detects a predetermined action of the external person detected by the first detection unit; a first authentication unit that authenticates whether the external person detected by the first detection unit is a pre-registered user of the vehicle, i.e., a registered user, based on a predetermined part of the external person detected by the first detection unit; and a door control unit that performs door opening control related to opening the vehicle door, wherein if the predetermined action detected by the second detection unit is the predetermined action of the external person authenticated by the first authentication unit as the registered user, the door control unit performs the door opening control in a first manner based on the predetermined action detected by the second detection unit.

[0011] Invention Effects

[0012] According to one aspect of the present invention, in the control of vehicle doors, it is possible to improve vehicle security while ensuring the convenience of vehicle users. Attached Figure Description

[0013] Figure 1 This is a diagram showing the structure of the control system.

[0014] Figure 2 This is a diagram showing the structure of the vehicle.

[0015] Figure 3 This is a diagram showing the main structural components of the control system.

[0016] Figure 4 This is a diagram illustrating an example of the databases stored in the second memory.

[0017] Figure 5 This is a flowchart illustrating the actions entering the ECU and area A-ECU.

[0018] Figure 6 It is a diagram used to illustrate the timing of actions in a control system.

[0019] Figure 7 This is a flowchart illustrating the actions entering the ECU and area A-ECU.

[0020] Figure 8 This is a flowchart illustrating the actions entering the ECU and area A-ECU.

[0021] Figure 9 This is a flowchart illustrating the actions entering the ECU and area A-ECU.

[0022] Figure 10 This is a flowchart illustrating the actions entering the ECU and area A-ECU.

[0023] Explanation of reference numerals in the attached figures

[0024] 1…Control system, 2…Central ECU, 3a…First communication line, 4c…Second communication line, 5…In-vehicle connection link, 7…Rear camera, 29…Area A-ECU, 33…Door locking mechanism, 36…Entry ECU, 38…Front camera, 39…Right-side camera, 40…Left-side camera, 41A…Driver's seat, 42…Door, 43…Camera (exterior image acquisition unit), 100…First processor, 101…First communication control unit, 102…Door control unit, 110…First memory, 111…First control program, 200…Second processor, 201…Second communication control unit, 202…Registration unit, 203… Approach detection department, 204…External person detection department (first detection department), 205…Approach method authentication department (second authentication department), 206…First gesture detection department (second detection department), 207…Face authentication department (first authentication department), 208…Disembarkation detection department, 209…Disembarkation person detection department (third detection department), 210…Second gesture detection department (fourth detection department), 220…Second memory, 221…Second control program, 222…Approach method database, 223…Face database, 224…First gesture database, 225…Second gesture database, 226…Third gesture database, V…Vehicle, VP…External person. Detailed Implementation

[0025] Figure 1 This is a diagram showing the vehicle's control system 1.

[0026] The control system 1 includes a central ECU (Electronic Control Unit) 2 for overall vehicle control and information processing. The central ECU 2 is connected to communication lines including first communication lines 3a and 3b and a second communication line 4c. The central ECU 2 functions as a gateway managing the transmission and reception of communication data between these communication lines. Furthermore, the central ECU 2 performs OTA (Over-The-Air) management. OTA management includes processing the download of updates to the vehicle's onboard devices from an external server, and the control related to applying the downloaded updates to the onboard devices.

[0027] The first communication lines 3a and 3b, and the second communication line 4c, are constructed from a bus that performs communication according to standards such as CAN (Controller Area Network) or Ethernet (registered trademark), or from communication lines that perform P2P (Peer-to-Peer) communication. Furthermore, the first communication lines 3a and 3b can be constructed from multiple communication lines performing communication according to the same standard, or from multiple communication lines performing communication according to different standards. The same applies to the second communication line 4c.

[0028] The first communication line 3a connects to the ICB (Infotainment Control Box) 6, speaker 8, microphone 9, and instrument panel 10 via in-vehicle connection link 5. The instrument panel 10 displays information about the vehicle's operating status, including vehicle speed.

[0029] Additionally, the in-vehicle connection link 5 connects a TCU (Telematics Control Unit) 12 and a touch panel 15. The TCU 12 is a wireless communication device that conforms to the communication standards of mobile communication systems. The touch panel 15 includes a display 16 and a touch sensor 17.

[0030] ICB6 is the IVI (In-Vehicle Infotainment) ECU. ICB6 uses speakers 8, microphones 9, and a touch panel 15 to provide various information and entertainment to the vehicle's occupants.

[0031] The in-vehicle connection link 5 consists of multiple communication transmission paths conforming to various communication standards. For example, the in-vehicle connection link 5 may also include multiple network transmission paths. In this case, the multiple network transmission paths may also be interconnected via a device with gateway functionality. Additionally, the in-vehicle connection link 5 may include a transmission path for P2P communication. In the network transmission paths, various communication buses conforming to various standards for network communication can be used. Examples of such standards include CAN, Ethernet, USB (Universal Serial Bus), LIN (Local Interconnect Network), and LVDS (Low Voltage Differential Signaling), but other standards may also be used.

[0032] A DMC (Driver Monitoring Camera) 18 for monitoring the driver is connected to the first communication line 3b.

[0033] Area A-ECU 29 is connected to the second communication line 4c. Area A-ECU 29 is connected to the lamp body 30 and the window motor 31. The lamp body 30 includes, for example, headlights, taillights, and turn signals. The window motor 31 opens and closes the windows. In addition, Area A-ECU 29 is connected to the door sensor 32, the door lock mechanism 33, and the ESL (Electronic Steering Lock) 34. The door sensor 32 detects door operations. The door lock mechanism 33 locks and unlocks the vehicle doors. Area A-ECU 29 is connected to the entry ECU 36. The entry ECU 36 is connected to an LF / RF antenna 37 that communicates wirelessly with the vehicle's electronic key. The electronic key is an electronic device with wireless communication capabilities, called a smart key or FOB key. The entry ECU 36 cooperates with other onboard ECUs to process user access from outside the vehicle to the control system 1, realizing the so-called smart entry action.

[0034] The ECU36 is also connected to a rear camera 7, a front camera 38, a right-side camera 39, and a left-side camera 40.

[0035] The following diagrams will use the "V" symbol to indicate the vehicle with control system 1.

[0036] Figure 2 This is a diagram showing the structure of vehicle V.

[0037] Figure 2 The vehicle V shown is a four-wheeled car with a seating capacity of 5. Vehicle V has a driver's seat 41A, a front passenger seat 41B, a rear right seat 41C, a rear center seat 41D, and a rear left seat 41E.

[0038] Vehicle V has a right front door 42A, a left front door 42B, a right rear door 42C, a left rear door 42D, and a tailgate 42E. Access to the passenger compartment of Vehicle V is possible by opening the right front door 42A, left front door 42B, right rear door 42C, and left rear door 42D. The tailgate 42E opens and closes the trunk of Vehicle V.

[0039] In the following description, without distinguishing between the right front door 42A, left front door 42B, right rear door 42C, left rear door 42D, and tailgate 42E, they are referred to as "doors" and marked with the figure mark "42".

[0040] Vehicle V is equipped with a door locking mechanism 33. The door locking mechanism 33 includes door locking mechanisms 33A, 33B, 33C, 33D, and 33E.

[0041] Door lock mechanism 33A unlocks and locks the right front door 42A. Door lock mechanism 33B unlocks and locks the left front door 42B. Door lock mechanism 33C unlocks and locks the right rear door 42C. Door lock mechanism 33D unlocks and locks the left rear door 42D. Door lock mechanism 33E unlocks and locks the tailgate 42E.

[0042] A front-facing camera 38 is installed at the front of vehicle V to capture images of the front of vehicle V. A rear-facing camera 7 is installed at the rear of vehicle V to capture images of the rear of vehicle V. A right-facing camera 39 is installed on the right side of vehicle V to capture images of the right side of vehicle V. A left-facing camera 40 is installed on the left side of vehicle V to capture images of the left side of vehicle V. The front-facing camera 38, rear-facing camera 7, right-facing camera 39, and left-facing camera 40 are exterior cameras that capture images of the exterior of vehicle V. In the following description, without distinguishing between the front-facing camera 38, rear-facing camera 7, right-facing camera 39, and left-facing camera 40, they are referred to as "cameras" and marked with the reference numeral "43". Camera 43 captures video of the exterior of vehicle V. The dynamic images captured by camera 43 include images of the exterior of vehicle V, i.e., exterior images.

[0043] Camera 43 is equivalent to the "exterior image acquisition unit" disclosed herein.

[0044] Figure 3 This is a diagram showing the main structural components of control system 1.

[0045] Area A-ECU29 is connected to the central ECU2. Additionally, Area A-ECU29 is connected to the door lock mechanism 33 and the entry ECU 36. The rear camera 7, front camera 38, right-side camera 39, and left-side camera 40 are connected to the entry ECU 36.

[0046] Region A-ECU29 includes a first processor 100 (CPU, Central Processing Unit, MPU, Microprocessor Unit), a first memory 110, and interface circuits for connecting devices, sensors, and other equipment.

[0047] The first memory 110 is a storage device that non-volatilely stores programs and data executed by the first processor 100. The first memory 110 is composed of a magnetic storage device, a semiconductor storage element such as flash memory (ROM), or other types of non-volatile storage devices. Alternatively, the first memory 110 may also include RAM (Random Access Memory) constituting the working area of ​​the first processor 100. Furthermore, the first memory 110 may also include non-volatile storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive). The first memory 110 stores data processed by the first processor 100 and a first control program 111 executed by the first processor 100.

[0048] The first processor 100 functions as the first communication control unit 101, the door control unit 102, and the door opening control prohibition unit 103 by reading and executing the first control program 111 stored in the first memory 110.

[0049] The first communication control unit 101 receives door control content information J2 from the input ECU 36. The door control content information J2 is information indicating the control content related to door 42.

[0050] The door control unit 102 performs door opening control. Door opening control refers to the control related to opening the door 42, including the unlocking of the door 42 and the opening action of the door 42. In this embodiment, unlocking the door 42 is exemplified as door opening control. The door control unit 102 performs door opening control on the door 42 as the controlled object based on the control content of the door control content information J2 received by the first communication control unit 101. Furthermore, in door opening control, the door control unit 102 outputs an unlock signal to the door lock mechanism 33 corresponding to the door 42 as the controlled object.

[0051] The door control unit 102 performs door closing control. Door closing control refers to the control related to closing the door 42, including locking the door 42 and closing the door 42. In this embodiment, locking the door 42 is exemplified as door closing control. The door control unit 102 performs door closing control on the door 42 as the controlled object based on the control content of the door control content information J2 received by the first communication control unit 101. The door control unit 102 outputs a locking signal to the door lock mechanism 33 during door closing control.

[0052] The door opening control prohibition unit 103 prevents the door control unit 102 from performing door opening control. When the first communication control unit 101 receives specific door control content information J2, the door opening control prohibition unit 103 prevents the door control unit 102 from performing door opening control. The door opening control prohibition unit 103 prevents the door control unit 102 from performing door opening control by outputting prohibition information to the door control unit 102. If the door control unit 102 receives prohibition information from the door opening control prohibition unit 103, it will not perform door opening control even if the first communication control unit 101 receives door control content information J2. If a predetermined trigger such as unlocking of the door 42 based on an electronic key occurs, the door opening control prohibition unit 103 cancels the prohibition of door opening control. The door opening control prohibition unit 103 outputs cancellation information to the door control unit 102, thereby canceling the prohibition of door opening control by the door control unit 102. If the door control unit 102 receives a stop information from the door opening control prohibition unit 103, it resumes performing door opening control.

[0053] The ECU 36 has interface circuits that connect to a second processor 200 (such as a CPU and MPU), a second memory 220, and devices such as sensors. Dynamic image data is input to the ECU 36 from the camera 43. This dynamic image data is the data of dynamic images captured by the camera 43.

[0054] The second memory 220 is a storage device that non-volatilely stores programs and data executed by the second processor 200. The second memory 220 is composed of a magnetic storage device, a semiconductor storage element such as flash memory (ROM), or other types of non-volatile storage devices. Alternatively, the second memory 220 may also include RAM constituting the working area of ​​the second processor 200. Furthermore, the second memory 220 may also include non-volatile storage devices such as HDDs and SSDs. The second memory 220 stores data processed by the second processor 200, a second control program 221 executed by the second processor 200, an proximity mode database 222, a face database 223, a first gesture database 224, a second gesture database 225, and a third gesture database 226, etc.

[0055] Reference Figure 4 The databases stored in the second memory 220 will be described.

[0056] Figure 4 This is a diagram illustrating an example of the databases stored in the second memory 220.

[0057] The approach mode database 222 contains information about the approach mode (hereinafter referred to as "approach mode") for each user's approach to vehicle V. Each record in the approach mode database 222 contains approach mode data SD. The approach mode data SD is data of the skeletal models BM of users of vehicle V approaching vehicle V, arranged in a time series. The skeletal model BM, for example, refers to a model that uses points to represent the positions of the head, shoulders, elbows, waist, knees, and ankles, connecting these points with lines to resemble a human posture.

[0058] The face database 223 contains face-related information for each user record of vehicle V. Each record in the face database 223 contains facial feature data (FAD) representing facial features. Facial features are numerical sets (NG) obtained by quantifying each feature point of the face. Examples of feature points include the positions of the eyes, nose, and mouth relative to the facial contour.

[0059] The face is equivalent to the "designated area" in this disclosure.

[0060] The first gesture database 224 contains gesture-related information for each gesture record. Each record in the first gesture database 224 contains gesture content information J1 and door control content information J2. The gesture content information J1 represents the gesture content itself.

[0061] exist Figure 4 In the example, in the first gesture database 224, gesture content information J1, which represents the gesture of "covering the face with the hand," corresponds to door control content information J2, which represents the control content of "preventing door opening." Additionally, in Figure 4 In the example, in the first gesture database 224, the gesture content information J1, which represents the gesture content of "raising the left hand", corresponds to the door control content information J2, which represents the control content of "unlocking the right rear door".

[0062] The gestures are equivalent to the “standard actions” of this disclosure.

[0063] The second gesture database 225 contains gesture-related information for each gesture record. In the second gesture database 225 and the first gesture database 224, the door control content corresponding to the gesture content differs, except for gesture content such as "covering the face with the hand". Each record in the second gesture database 225 contains gesture content information J1 and door control content information J2.

[0064] exist Figure 4In the example, in the second gesture database 225, gesture content information J1, which represents the gesture of "covering the face with the hand," corresponds to door control content information J2, which represents the control content of "preventing door opening." Additionally, in Figure 4 In the example, in the second gesture database 225, gesture content information J1, which represents a gesture such as "raise your left hand", corresponds to door control content information J2, which represents a control such as "unlock all doors".

[0065] The third gesture database 226 records gesture-related information for each gesture. Unlike the first gesture database 224 and the second gesture database 225, the third gesture database 226 records door control content information J2, which represents control content related to door closing control. That is, the first and second gesture databases 224 and 225 record door control content information J2, which represents control content related to door opening control. Each record in the third gesture database 226 contains gesture content information J1 and door control content information J2.

[0066] exist Figure 4 In the example, in the third gesture database 226, the gesture content information J1, which represents the gesture of "raising the right hand", corresponds to the door control content information J2, which represents the control content of "locking all the doors".

[0067] Return to Figure 3 As explained, the second processor 200 reads and executes the second control program 221 stored in the second memory 220, and functions as the second communication control unit 201, registration unit 202, proximity detection unit 203, vehicle exterior person detection unit 204, proximity method authentication unit 205, first gesture detection unit 206, face authentication unit 207, vehicle exit detection unit 208, vehicle exit person detection unit 209, and second gesture detection unit 210.

[0068] The external person detection unit 204 is equivalent to the "first detection unit" of this disclosure. The approach method authentication unit 205 is equivalent to the "second authentication unit" of this disclosure. The first gesture detection unit 206 is equivalent to the "second detection unit" of this disclosure. The face authentication unit 207 is equivalent to the "first authentication unit" of this disclosure. The exiting person detection unit 209 is equivalent to the "third detection unit" of this disclosure. The second gesture detection unit 210 is equivalent to the "fourth detection unit" of this disclosure.

[0069] The second communication control unit 201 communicates with area A-ECU 29.

[0070] Registration unit 202 records the approach mode data SD of the user of vehicle V in the approach mode database 222. Registration unit 202 generates a skeletal model BM for each exterior image of the dynamic images captured by camera 43, and generates approach mode data SD by arranging the generated skeletal models in a time sequence, and records it in the approach mode database 222. Furthermore, the skeletal model BM is generated using existing skeletal detection or existing skeletal detection technology. Thus, by recording the approach mode data SD in the approach mode database 222, the user of vehicle V can register their approach posture with vehicle V by having camera 43 capture their approach posture.

[0071] Registration unit 202 records the facial feature data (FAD) of the user of vehicle V in facial database 223. Registration unit 202 generates facial feature data (FAD) based on dynamic images captured by camera 43 and records it in facial database 223. Thus, by recording facial feature data (FAD) in facial database 223, the user of vehicle V can register their face with vehicle V by having camera 43 capture their face.

[0072] The registration unit 202 records the combination of gesture content and control content in the first gesture database 224. The registration unit 202 identifies the gesture content of the user of vehicle V based on the dynamic image captured by camera 43 using methods such as image analysis. Then, the registration unit 202 records the door control content information J2, representing the control content specified by the user of vehicle V, and the gesture content information J1 represented by the identified gesture content in the first gesture database 224. In this way, the registration unit 202 records the combination of gesture content and control content in the first gesture database 224, allowing the user of vehicle V to register the desired gesture for performing the desired door opening control in vehicle V. The registration unit 202 similarly records the combination of gesture content and control content in the second gesture database 225 and the third gesture database 226.

[0073] The proximity detection unit 203 detects the approach of a person towards the vehicle V. For example, the proximity detection unit 203 detects the approach of a person towards the vehicle V based on the detection values ​​of the sonar sensor equipped on the vehicle V. Alternatively, for example, the proximity detection unit 203 detects the approach of a person towards the vehicle V by using the detection of an electronic key via the LF / RF antenna 37. Furthermore, the detection method of the proximity detection unit 203 is merely one example and is not limited to the above example. When the proximity detection unit 203 detects the approach of a person towards the vehicle V, the camera 43 of this embodiment is activated and begins to capture images. The proximity detection unit 203 may also keep the camera 43 constantly activated, detecting the approach of a person towards the vehicle V based on the image captured by the camera 43. Alternatively, the proximity detection unit 203 may also use other sensors, such as a NIR (Near Infrared) sensor (not shown), to detect approach.

[0074] The vehicle exterior person detection unit 204 detects vehicle exterior persons (VPs) existing outside the vehicle V based on the vehicle exterior images captured by the camera 43. Specifically, the vehicle exterior person detection unit 204 detects images of vehicle exterior persons (VPs) from the vehicle exterior images using methods such as pattern matching, thereby detecting the vehicle exterior person (VP).

[0075] The approach method authentication unit 205 authenticates whether the person outside the vehicle (VP) detected by the person outside the vehicle (VP) is a registered user based on the approach method detected by the person outside the vehicle (VP) detected by the person outside the vehicle (VP) 204. A registered user refers to a person who is registered with the vehicle V as a user of the vehicle V.

[0076] The approach mode authentication unit 205 generates approach mode data SD for the external person VP detected by the external person detection unit 204. Specifically, the approach mode authentication unit 205 generates a skeletal model BM of the external person VP detected by the external person detection unit 204 for each external image of the dynamic image captured by the camera 43, and generates approach mode data SD obtained by arranging the generated skeletal model in a time sequence.

[0077] The proximity authentication unit 205 compares the time-varying skeletal model BM shown in the generated proximity data SD with the time-varying skeletal model BM shown in the proximity data SD in the proximity database 222. If the proximity database 222 stores proximity data SD whose time-varying consistency with the generated skeletal model BM is above a predetermined value, the proximity authentication unit 205 authenticates the external person VP detected by the external person detection unit 204 as a registered user. On the other hand, if the proximity database 222 does not store proximity data SD whose time-varying consistency with the generated skeletal model BM is above a predetermined value, the proximity authentication unit 205 authenticates the external person VP detected by the external person detection unit 204 as not a registered user.

[0078] The first gesture detection unit 206 detects the gestures of the external person (VP) detected by the external person detection unit 204. More specifically, after authentication by the proximity authentication unit 205, the first gesture detection unit 206 detects the gestures of the external person (VP) authenticated as a registered user by the proximity authentication unit 205. Additionally, after authentication by the face authentication unit 207, the first gesture detection unit 206 detects the gestures of the external person (VP) authenticated as a registered user by the face authentication unit 207. The first gesture detection unit 206 tracks the image of the external person (VP) authenticated as a registered user in the dynamic image captured by the camera 43. The first gesture detection unit 206 performs image analysis on the tracked image of the external person (VP) and extracts the features of the external person (VP)'s movements (e.g., motion vectors). Then, based on the extracted features of the external person (VP)'s movements, the first gesture detection unit 206 identifies the content of the external person (VP)'s movements. Then, if the action content of the person VP outside the vehicle is the gesture content recorded in the first gesture database 224 and the second gesture database 225, the first gesture detection unit 206 determines that the gesture of the person VP outside the vehicle has been detected.

[0079] In the scenario described later, the first gesture detection unit 206 detects gestures from a non-registered user (i.e., a user of vehicle V other than a registered user). The first gesture detection unit 206 tracks an image of a person (VP) outside the vehicle who is a non-registered user in the dynamic image captured by the camera 43. The first gesture detection unit 206 performs image analysis on the tracked image of the person (VP) outside the vehicle and extracts features of the person (VP)'s movements. Then, based on the extracted features of the person (VP)'s movements, the first gesture detection unit 206 identifies the content of the person (VP)'s movements. If the identified content of the person (VP)'s movements is a gesture recorded in the first gesture database 224 and the second gesture database 225, the first gesture detection unit 206 determines that a gesture from a non-registered user has been detected.

[0080] The face authentication unit 207 authenticates whether the person outside the vehicle (VP) authenticated as a registered user by the proximity authentication unit 205 is indeed a registered user. That is, the control system 1 of this embodiment performs Level 2 authentication. The face authentication unit 207 authenticates whether the person outside the vehicle (VP) is a registered user based on their face. The face authentication unit 207 extracts an image of the face of the person outside the vehicle (VP) from the dynamic image captured by the camera 43, and calculates facial feature values ​​based on the extracted face image. The face authentication unit 207 compares the calculated facial feature values ​​with the facial feature value data FAD recorded in the face database 223, and determines whether the facial feature value data FAD with a consistency rate of at least a threshold is included in the face database 223. If the face authentication unit 207 determines that the facial feature value data FAD with a consistency rate of at least a threshold is included in the face database 223, the face authentication unit 207 authenticates the person outside the vehicle (VP) authenticated as a registered user by the proximity authentication unit 205 as a registered user. On the other hand, if the facial feature data FAD, which has a consistency rate with the calculated facial feature data above the threshold, is not included in the facial database 223, the facial authentication unit 207 authenticates that the person outside the vehicle VP, who is registered as the registered user by the proximity authentication unit 205, is not the registered user.

[0081] The exit detection unit 208 detects whether an occupant has exited the vehicle V. For example, if the ignition power to the vehicle V is disconnected and the rear door 42 is open, the exit detection unit 208 detects that an occupant has exited the vehicle V. Furthermore, the detection method of the exit detection unit 208 is merely one example and is not limited to the above example.

[0082] The disembarking person detection unit 209 detects people who disembark from vehicle V. The disembarking person detection unit 209 also detects people outside vehicle V who disembark and leave vehicle V as disembarking persons. For example, the disembarking person detection unit 209 detects people outside vehicle V whose image area decreases from a predetermined value in the dynamic image captured by camera 43 as disembarking persons.

[0083] The second gesture detection unit 210 detects the gestures of the disembarking person detected by the disembarking person detection unit 209. The second gesture detection unit 210 tracks the image of the disembarking person detected by the disembarking person detection unit 209 within the dynamic image captured by the camera 43. The second gesture detection unit 210 performs image analysis on the tracked image of the disembarking person, extracting the features of the disembarking person's movements. Then, based on the extracted features of the disembarking person's movements, the second gesture detection unit 210 identifies the content of the disembarking person's movements. Finally, if the identified disembarking person's movements match the gesture content recorded in the third gesture database 226, the second gesture detection unit 210 determines that a gesture of the disembarking person has been detected.

[0084] Next, the operation of each part of the control system 1 will be explained.

[0085] First, let me explain the operation of the control system 1 when the external person detection unit 204 detects an external person VP.

[0086] Figure 5 This is a flowchart illustrating the actions entering ECU36 and region A-ECU29. Figure 5 In the flowchart, FA represents the action entering ECU36, and FB represents the action in area A-ECU29.

[0087] As shown in flowchart FA, the approach authentication unit 205 authenticates whether the external person VP detected by the external person detection unit 204 is a registered user (step SA1).

[0088] Next, the first gesture detection unit 206 determines whether the user was authenticated as a registered user in step SA1 (step SA2). If the user is determined not to be a registered user (step SA2: no), the second processor 200 terminates the process.

[0089] On the other hand, when the first gesture detection unit 206 determines that the user is a registered user (step SA2: Yes), it begins to detect gestures (step SA3).

[0090] Next, the face authentication unit 207 verifies whether the external person VP of the registered user is the registered user (step SA4) by the proximity authentication unit 205.

[0091] Next, the first gesture detection unit 206 determines whether the user has been authenticated as a registered user in step SA4 (step SA5).

[0092] If the first gesture detection unit 206 determines that the authentication is not that of a registered user (step SA5: No), the second processor 200 ends the process.

[0093] On the other hand, if the first gesture detection unit 206 determines that the user is a registered user (step SA5: Yes), it determines whether a gesture is detected before the face authentication unit 207 authenticates (step SA6).

[0094] If the first gesture detection unit 206 determines that a gesture was detected before the face authentication unit 207 authenticates (step SA6: Yes), it retrieves the door control content information J2 corresponding to the detected gesture from the first gesture database 224 (step SA7).

[0095] Next, the second communication control unit 201 sends the door control content information J2 obtained in step SA7 to area A-ECU29 (step SA8).

[0096] Returning to the explanation of step SA6, if the first gesture detection unit 206 determines that no gesture was detected before the face authentication unit 207 authenticates (step SA6: No), it determines whether a gesture was detected (step SA9). Furthermore, if the second processor 200 does not make an affirmative determination within a predetermined period from the start of the determination in step SA9, it may terminate this process.

[0097] When the first gesture detection unit 206 determines that a gesture has been detected (step SA9: Yes), it retrieves the door control content information J2 corresponding to the detected gesture from the first gesture database 224 (step SA7).

[0098] Next, the second communication control unit 201 sends the door control content information J2 obtained in step SA7 to area A-ECU29 (step SA8).

[0099] As shown in flowchart FB, the first communication control unit 101 of region A-ECU29 receives gate control content information J2 from the entry ECU36 (step SB1).

[0100] The first communication control unit 101 determines whether the control content shown in the received door control content information J2 is content that prohibits opening the door (step SB2). Furthermore, in Figure 4 In the example, the door control content information J2, which is the content of the prohibition of opening the door control, is the door control content information J2 that represents the control content such as "prohibition of opening the door control".

[0101] If the first communication control unit 101 determines that the control content shown in the door control content information J2 is content that prohibits door opening control (step SB2: Yes), the door opening control prohibition unit 103 prohibits the execution of door opening control (step SB3).

[0102] On the other hand, if the first communication control unit 101 determines that the control content shown in the door control content information J2 is not the content that prohibits door opening control (step SB2: no), the door control unit 102 performs door opening control according to the control content shown in the door control content information J2 received in step SB1 (step SB4).

[0103] Next, refer to Figure 6 The timing of the actions of control system 1 is explained.

[0104] Figure 6 This is a diagram used to illustrate the timing of the actions of control system 1.

[0105] Figure 6 This illustrates a scenario where a person (VP) outside the vehicle, acting as a registered user, unlocks door 42 of vehicle V via a gesture, intending to get into vehicle V. Additionally, Figure 6 This shows the situation where the first gesture detection unit 206 detects the gestures of the person VP outside the vehicle before the face authentication unit 207 performs the authentication.

[0106] exist Figure 6 The diagram shows four positions: I1, I2, I3, and I4. Position I1 is the position farthest from vehicle V among positions I1, I2, I3, and I4. Position I2 is closer to vehicle V than position I1, but farther from vehicle V than positions I3 and I4. Position I3 is closer to vehicle V than positions I1 and I2, but farther from vehicle V than position I4. Position I4 is the position closest to vehicle V among positions I1, I2, I3, and I4.

[0107] exist Figure 6 In the middle, the proximity detection unit 203 detects the approach of a person VP outside the vehicle at time T1. Furthermore, in Figure 6 In the process, the person outside the vehicle VP is located at position I1 at time T1. When the proximity detection unit 203 detects the approach of the person outside the vehicle VP at time T1, the person outside the vehicle detection unit 204 performs the detection of the person outside the vehicle VP after time T1.

[0108] exist Figure 6 In the process, the proximity authentication department 205 authenticates the external person (VP) as a registered user at time T2. Furthermore, in... Figure 6 In the event that the person outside the vehicle (VP) is located at position I2 at time T2, when the approach authentication unit 205 authenticates the person outside the vehicle (VP) as a registered user at time T2, the first gesture detection unit 206 performs gesture detection after time T2.

[0109] exist Figure 6 In the middle, the first gesture detection unit 206 detects the gesture of the person VP outside the vehicle at time T3. Furthermore, in Figure 6 In the middle, the person VP outside the car is located at position I3 at time T3.

[0110] exist Figure 6 In the middle, the facial recognition department 207 at time T4 authenticated the person outside the car as a registered user. Furthermore, in Figure 6 At time T4, the person outside the vehicle (VP) is located at position I4. If the face authentication unit 207 authenticates the person outside the vehicle (VP) as a registered user at time T4, then at time T4, the door control unit 102 performs door opening control corresponding to the gesture detected at time T3.

[0111] Next, the operation of each part of the control system 1 when the vehicle external person detection unit 204 detects multiple vehicle external persons (VPs) will be explained.

[0112] Figure 7 This is a flowchart illustrating the actions entering ECU36 and region A-ECU29. Figure 7 In the flowchart, FC represents the action entering ECU36, and FD represents the action in area A-ECU29. Figure 7 In the flowchart, for Figure 5 The same steps are labeled with the same step number and their detailed descriptions are omitted.

[0113] As shown in flowchart FC, the proximity authentication unit 205 authenticates each external person VP detected by the external person detection unit 204 to determine whether they are a registered user (step SC1).

[0114] Next, based on the authentication result of step SC1, the proximity authentication unit 205 determines whether the multiple external persons VPs detected by the external persons detection unit 204 include only registered users, only non-registered users, or both registered and non-registered users (step SC2).

[0115] If it is determined that the process only includes unregistered users (step SC2: only unregistered users), the second processor 200 ends the process.

[0116] If the system determines that the data only includes registered users (step SC2: only registered users), the second processor 200 performs the corresponding processing (step SC3). Here, the corresponding processing is... Figure 5 The processing steps SA3 to SA8 are described in the text.

[0117] If the system determines that the user includes both registered and unregistered users (step SC2: registered and unregistered users), the first gesture detection unit 206 begins to detect the gestures of registered users (step SC4).

[0118] The face authentication unit 207 verifies whether the external person VP, who has been authenticated as a registered user by the proximity authentication unit 205, is indeed a registered user (step SC5).

[0119] Next, the first gesture detection unit 206 determines whether all external persons (VPs) who were authenticated as authentication objects in step SC5 are not registered users (step SC6).

[0120] If all external persons (VPs) identified as being to be authenticated are not registered users (step SC6: No), the second processor 200 terminates this process.

[0121] On the other hand, if at least a portion of all external persons (VPs) identified as authentication targets are authenticated as registered users (step SC6: Yes), the first gesture detection unit 206 determines whether a gesture is detected before the face authentication unit 207 authenticates (step SC7).

[0122] If the first gesture detection unit 206 determines that a gesture was detected before the face authentication unit 207 authenticates (step SC7: Yes), it determines whether the detected gesture is the gesture of a registered user who has been authenticated by the face authentication unit 207 (step SC8).

[0123] If the detected gesture is determined to be not the gesture of a registered user certified by the face authentication unit 207 (step SC8: No), the second processor 200 ends the process.

[0124] On the other hand, when the first gesture detection unit 206 determines that the detected gesture is the gesture of a registered user certified by the face authentication unit 207 (step SC8: Yes), it obtains the door control content information J2 corresponding to the detected gesture from the second gesture database 225 (step SC9).

[0125] Next, the second communication control unit 201 sends the door control content information J2 obtained in step SC9 to area A-ECU29 (step SC10).

[0126] Returning to the explanation of step SC7, if the first gesture detection unit 206 determines that no gesture was detected before the face authentication unit 207 authenticates (step SC7: No), it determines whether a gesture was detected (step SC11). Furthermore, if the second processor 200 does not make an affirmative determination within a predetermined period from the start of the determination in step SC11, it may terminate this process.

[0127] When the first gesture detection unit 206 determines that a gesture has been detected (step SC11: Yes), it retrieves the door control content information J2 corresponding to the detected gesture from the second gesture database 225 (step SC9).

[0128] Next, the second communication control unit 201 sends the door control content information J2 obtained in step SC9 to area A-ECU29 (step SC10).

[0129] As described above, when the external person detection unit 204 detects multiple external persons (VPs) including registered users and non-registered users, the entry ECU 36 sends the door control content information J2 obtained from the second gesture database 225 to the area A-ECU 29. On the other hand, when the external person detection unit 204 detects only a registered user as an external person (VP), the entry ECU 36 sends the door control content information J2 obtained from the first gesture database 224 to the area A-ECU 29. As described above, even if the gesture content is the same in the first gesture database 224 and the second gesture database 225, the corresponding door control content is different. Therefore, even if the gesture performed by the registered user is the same, the door control unit 102 can control the door opening differently depending on whether the non-registered user is present with the registered user. That is, the door control unit 102 can perform door opening control in a first manner when the non-registered user is not present with the registered user, and can perform door opening control in a second manner when the non-registered user is present with the registered user.

[0130] Next, the actions of each part of the control system 1 will be explained when, after the external person detection unit 204 detects multiple external persons (VPs), including registered users and unregistered users, and the door opening control is performed based on the gesture of the registered user, the actions of each part of the control system 1 will be explained. That is, the actions of each part of the control system 1 will be explained when the external person detection unit 204 detects multiple external persons (VPs), including registered users and unregistered users, and the door opening control is performed based on the gesture of the registered user. Figure 7 The actions of each part of the control system 1 after steps SC10 and SB4 will be explained.

[0131] Figure 8 This is a flowchart illustrating the actions entering ECU36 and region A-ECU29. Figure 8 In the flowchart, FE represents the action entering ECU36, and FF represents the action in area A-ECU29.

[0132] exist Figure 8 In the flowchart, for the... Figure 5 The same steps are labeled with the same step number and their detailed descriptions are omitted.

[0133] As shown in flowchart FE, the first gesture detection unit 206 determines whether a gesture from a non-registered user has been detected (step SE1). Furthermore, if the second processor 200 does not make an affirmative determination within a predetermined period from the start of the determination in step SE1, it may terminate the process.

[0134] When the first gesture detection unit 206 determines that a gesture from an unregistered user has been detected (step SE1: Yes), it retrieves the door control content information J2 corresponding to the detected gesture from the first gesture database 224 (step SE2). Alternatively, in step SE2, the first gesture detection unit 206 may also retrieve the door control content information J2 from the second gesture database 225.

[0135] Next, the second communication control unit 201 sends the door control content information J2 obtained in step SE2 to area A-ECU29 (step SE3).

[0136] When the external person / object detection unit 204 detects multiple external person / object (VP) devices including multiple registered users, each of the multiple registered users may perform a gesture simultaneously or at short time intervals. Therefore, when the external person / object detection unit 204 detects multiple external person / object (VP) devices including multiple registered users, the control system 1 can also execute... Figure 9 The action.

[0137] Figure 9 This is a flowchart illustrating the actions entering ECU36 and region A-ECU29. Figure 9 In the flowchart, FG represents the action entering ECU36, and FH represents the action in area A-ECU29.

[0138] exist Figure 9 In the flowchart, for the... Figure 5 The same steps are labeled with the same step number and their detailed descriptions are omitted.

[0139] exist Figure 9 At the start time of the flowchart shown, the face authentication unit 207 authenticates two or more of the multiple external person VPs detected by the external person detection unit 204 as registered users.

[0140] As shown in flowchart FG, the first gesture detection unit 206 determines whether a gesture is detected before the face authentication unit 207 authenticates (step SG1).

[0141] If the first gesture detection unit 206 determines that a gesture was detected before the face authentication unit 207 authenticates (step SG1: Yes), it determines whether the gestures of multiple registered users have been detected (step SG2).

[0142] When the first gesture detection unit 206 determines that multiple registered users' gestures have been detected (step SG2: Yes), it acquires door control content information J2 corresponding to the gesture of the person outside the vehicle VP closest to the driver's seat 41A among the detected gestures (step SG3). Here, the person outside the vehicle VP closest to the driver's seat 41A refers to the person outside the vehicle VP with the smallest distance from the vehicle V and located closest to the driver's seat 41A in the longitudinal direction of the vehicle V. The first gesture detection unit 206 identifies the person outside the vehicle VP closest to the driver's seat based on the image area of ​​the person outside the vehicle VP in the external image captured by the camera 43 and the position of the person outside the vehicle VP.

[0143] Furthermore, in step SG3, if all of the multiple external person VPs detected by the external person detection unit 204 are registered users, the first gesture detection unit 206 obtains the door control content information J2 from the first gesture database 224. Additionally, in step SG3, if some of the multiple external person VPs detected by the external person detection unit 204 are registered users, the first gesture detection unit 206 obtains the door control content information J2 from the second gesture database 225.

[0144] When the door control content information J2 is obtained in step SG3, the second communication control unit 201 sends the obtained door control content information J2 to area A-ECU29 (step SG4).

[0145] Returning to the explanation of step SG2, if the first gesture detection unit 206 determines that no gestures of multiple registered users have been detected (step SG2: No), it obtains the door control content information J2 corresponding to the detected gestures (step SG5).

[0146] Furthermore, in step SG5, if all of the multiple external person VPs detected by the external person detection unit 204 are registered users, the first gesture detection unit 206 obtains the door control content information J2 from the first gesture database 224. Additionally, in step SG5, if some of the multiple external person VPs detected by the external person detection unit 204 are registered users, the first gesture detection unit 206 obtains the door control content information J2 from the second gesture database 225.

[0147] Next, the second communication control unit 201 sends the door control content information J2 obtained in step SG5 to area A-ECU29 (step SG6).

[0148] Returning to the explanation of step SG1, if the first gesture detection unit 206 determines that no gesture was detected before the face authentication unit 207 authenticates (step SG1: No), it determines whether a gesture was detected (step SG7). Furthermore, if the second processor 200 does not make an affirmative determination within a predetermined period from the start of the determination in step SG7, it may terminate this process.

[0149] If the first gesture detection unit 206 determines that a gesture has been detected (step SG7: Yes), the processing is transferred to step SG2.

[0150] Next, the operation of each part of the control system 1 related to door closing control will be explained.

[0151] Figure 10 This is a flowchart illustrating the actions entering ECU36 and region A-ECU29.

[0152] exist Figure 10 In the flowchart, FI represents the action entering ECU36, and FJ represents the action in area A-ECU29.

[0153] As shown in flowchart FI, the disembarking passenger detection unit 209 determines whether the disembarking detection unit 208 has detected the passenger disembarking (step SI1).

[0154] If the disembarking personnel detection unit 209 determines that the disembarking personnel detection unit 208 has detected a passenger disembarking (step SI1: Yes), it begins to detect the disembarking personnel (step SI2).

[0155] Next, the second gesture detection unit 210 determines whether the disembarking person detection unit 209 has detected a disembarking person (step SI3). Furthermore, if the second processor 200 does not make an affirmative determination within a predetermined period from the start of step SI3, it may terminate this process.

[0156] If the second gesture detection unit 210 determines that the disembarking person detection unit 209 has detected a disembarking person (step SI3: Yes), it begins to detect the gesture of the disembarking person (step SI4).

[0157] The second gesture detection unit 210 determines whether a gesture has been detected (step SI5). Furthermore, if the second processor 200 does not make an affirmative determination within a predetermined period from the start of the determination in step SI5, it may terminate the process.

[0158] When the second gesture detection unit 210 determines that a gesture has been detected (step SI5: Yes), it retrieves the door control content information J2 corresponding to the gesture content of the detected gesture from the third gesture database 226 (step SI6).

[0159] Next, the second communication control unit 201 sends the door control content information J2 obtained in step SI6 to area A-ECU29 (step SI7).

[0160] As shown in flowchart FJ, the first communication control unit 101 of region A-ECU29 receives gate control content information J2 from the entry ECU36 (step SJ1).

[0161] Next, the door control unit 102 performs door closing control according to the door control content shown in the door control content information J2 received in step SJ1 (step SJ2).

[0162] The above-described implementation is merely one approach, which can be arbitrarily modified and applied.

[0163] In the above embodiment, unlocking of door 42 is exemplified as door opening control. However, door opening control is not limited to unlocking of door 42; if door 42 is an electric door, it could also be the opening action of door 42. Furthermore, if door 42 is an electric door and door opening control is the opening action of door 42, the control system 1 could also be a structure that allows the user to specify the degree of opening of door 42 based on gestures.

[0164] In the above embodiment, locking the door 42 is exemplified as a door closing control. However, the door closing control is not limited to locking the door 42; if the door 42 is an electric door, it can also be a closing action of the door 42. Furthermore, if the door 42 is an electric door and the door closing control is a closing action of the door 42, the control system 1 can also be a structure that allows the user to specify the degree of closure of the door 42 based on gestures.

[0165] In other embodiments, if the proximity authentication unit 205 authenticates the external person VP as a registered user, the area A-ECU 29 may also illuminate the lamp 30. The illuminated lamp 30 may be... Figure 1 The description may include any one or any combination of lamps listed in the instructions, or may include... Figure 1 Other lights not listed in the description (e.g., lights illuminating the ground opposite the underside of vehicle V).

[0166] In the above-described embodiment, when multiple registered users perform gestures, the gesture of the registered user closest to the driver's seat 41A takes priority. In other embodiments, when multiple registered users perform gestures and the door 42 being controlled does not conflict among the registered users, multiple gesture-related controls can be performed simultaneously.

[0167] In the above embodiments, gestures are exemplified as "prescribed actions" of this disclosure. However, the "prescribed actions" of this disclosure are not limited to gestures; for example, they may also be facial expressions.

[0168] In the above embodiments, the face is exemplified as a "defined part" of this disclosure. However, the "defined part" of this disclosure is not limited to the face; it may be a part constituting the face, or it may be a part other than the face, such as the hands or feet.

[0169] In the above embodiment, the ECU having the door control unit 102 and the door opening control prohibition unit 103 is described as region A-ECU29. However, the ECU having these functional units may also be designated as the central ECU2.

[0170] The first processor 100 and the second processor 200 can be composed of multiple processors or a single processor. These processors can also be hardware programmed to implement the aforementioned functionalities. In this case, these processors are, for example, composed of an ASIC (Application Specific Integrated Circuit) or a FPGA (Field Programmable Gate Array).

[0171] in addition, Figure 1 The structure of each part of the control system 1 shown is an example, and the specific installation method is not particularly limited. That is, it is not necessarily necessary to install hardware corresponding to each part; of course, it can also be configured so that the functions of each part are implemented by a processor executing a program. In addition, in the above embodiment, a part of the function implemented by software can be implemented as hardware, or a part of the function implemented by hardware can be implemented by software.

[0172] in addition, Figure 5 , Figures 7-9 The steps shown are divided based on the main processing content, and this invention is not limited to the method or name of the processing unit division. Depending on the processing content, it can also be divided into more step units. Alternatively, it can be divided so that one step unit contains more processes. Furthermore, the order of these steps can be appropriately changed without affecting the spirit of this invention.

[0173] Furthermore, when using a processor to implement the control method of the control system 1 described above, the program executed by the processor can also be configured as a recording medium or a transmission medium for transmitting the program. That is, the first control program 111 can also be implemented with the first control program 111 recorded on a removable information recording medium. Examples of information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, USB (Universal Serial Bus) memory, SSD (Solid State Drive) and other semiconductor storage devices, but other recording media can also be used. In addition, the second control program 221, like the first control program 111, can also be implemented with the second control program 221 recorded on a removable information recording medium.

[0174] The above implementation supports the following structures.

[0175] (Structure 1)

[0176] A control system for a vehicle includes: an exterior image acquisition unit that acquires exterior images as external images of the vehicle; a first detection unit that detects persons outside the vehicle based on the exterior images acquired by the exterior image acquisition unit; a second detection unit that detects a predetermined action of the person outside the vehicle detected by the first detection unit; a first authentication unit that authenticates whether the person outside the vehicle detected by the first detection unit is a pre-registered user of the vehicle, i.e., a registered user, based on a predetermined part of the person outside the vehicle detected by the first detection unit; and a door control unit that performs door opening control related to opening the vehicle door, wherein if the predetermined action detected by the second detection unit is the predetermined action of the person outside the vehicle authenticated by the first authentication unit as the registered user, the door control unit performs the door opening control in a first manner based on the predetermined action detected by the second detection unit.

[0177] According to the control system of Structure 1, door opening control is performed based on the prescribed actions of persons outside the vehicle who are certified as registered users, thus preventing door opening control from being performed due to prescribed actions of third parties. Therefore, in vehicle door control, vehicle safety can be improved while ensuring the convenience of vehicle users.

[0178] (Structure 2)

[0179] The control system described in Structure 1 includes a second authentication unit. This second authentication unit authenticates whether the person outside the vehicle detected by the first detection unit is a registered user based on the approach method of the person approaching the vehicle as detected by the first detection unit. The second detection unit detects a predetermined action of the person outside the vehicle who is authenticated as a registered user by the second authentication unit. The first authentication unit authenticates whether the person outside the vehicle who is authenticated as a registered user by the second authentication unit is a registered user based on the face of the person outside the vehicle. If the first authentication unit authenticates the person as a registered user when the second detection unit detects the predetermined action, the door control unit performs the door opening control at the moment of authentication by the first authentication unit.

[0180] According to the control system of Structure 2, door opening control is performed based on the prescribed actions of an external person after passing two-stage authentication, thus further improving vehicle safety. Furthermore, by configuring the system to detect the prescribed actions before facial recognition authentication, door opening control can be performed quickly after facial recognition authentication. Therefore, in vehicle door control, both user convenience and vehicle safety can be further improved.

[0181] (Structure 3)

[0182] In the control system described in Structure 1 or Structure 2, if the first detection unit detects multiple external persons, including users of the vehicle who are not the registered user (i.e., non-registered users) and the registered user, the door control unit performs the door opening control in a second manner based on the predetermined action detected by the second detection unit.

[0183] According to the control system of Structure 3, the door opening control method can be different depending on whether a non-registered user is present along with the registered user, thus enabling door opening control tailored to the vehicle's users. Therefore, the convenience of vehicle users can be further improved in the control of vehicle doors.

[0184] (Structure 4)

[0185] In the control system described in structure 3, if the door opening control is performed based on the prescribed action of the person outside the vehicle who is certified as the registered user by the first authentication unit, and the second detection unit detects the prescribed action of the non-registered user, the door control unit performs the door opening control based on the prescribed action of the non-registered user.

[0186] According to the control system of Structure 4, after the door opening control is performed based on the prescribed actions of the registered user, non-registered users can also open the vehicle doors through prescribed actions. Therefore, in the control of the vehicle doors, the convenience of the vehicle users can be further improved while ensuring vehicle safety.

[0187] (Structure 5)

[0188] The control system according to any one of structures 2 to 4, wherein the prescribed action is a gesture for the vehicle, the control system includes a door opening control prohibition unit, and when the gesture detected by the second detection unit is a gesture of covering the face with a hand, the door opening control prohibition unit prevents the door control unit from performing the door opening control.

[0189] According to the control system of Structure 5, by having a structure that does not perform door opening control based on a predetermined action, it is possible to respond to situations where the user does not wish to perform door opening control, thereby further improving the convenience for vehicle users. Furthermore, in the structure that authenticates whether someone is a registered user via facial recognition, by setting the gesture for not performing door opening control to a gesture of covering the face with the hand, registered users can easily understand what the gesture for not performing door opening control is.

[0190] (Structure 6)

[0191] The control system according to any one of structures 1 to 5, wherein, when the second detection unit detects a plurality of the prescribed actions, and the plurality of the prescribed actions detected by the second detection unit are the prescribed actions of a plurality of external persons who are certified by the first authentication unit as the registered user, the door control unit performs the door opening control based on the prescribed action of the external person closest to the driver's seat of the vehicle.

[0192] According to the control system of structure 6, the prescribed actions of persons outside the vehicle who are more likely to be the driver of the vehicle can be prioritized, thus enabling door opening control as intended by such persons. Therefore, the convenience of vehicle users can be further improved in vehicle door control.

[0193] (Structure 7)

[0194] The control system according to any one of structures 1 to 6, wherein the control system comprises: a third detection unit that detects a person getting off the vehicle based on an exterior image acquired by the exterior image acquisition unit; and a fourth detection unit that detects a predetermined action of the person getting off the vehicle detected by the third detection unit, and the door control unit performs door closing control related to closing the door of the vehicle based on the predetermined action detected by the fourth detection unit.

[0195] According to the control system of structure 7, door closing can be performed without authentication when exiting the vehicle, thus eliminating the need to turn back specifically to close the door after exiting. Therefore, the convenience of vehicle users can be further improved in vehicle door control.

[0196] (Structure 8)

[0197] A control method for a control system, specifically for a vehicle control system, includes: acquiring an exterior image of the vehicle; detecting external persons present outside the vehicle based on the acquired exterior image; detecting a predetermined action of the detected external person; authenticating whether the detected external person is a pre-registered user of the vehicle (i.e., a registered user) based on a predetermined part of the detected external person; and, if the detected predetermined action is the predetermined action of the external person authenticated as a registered user, performing door opening control related to opening the vehicle door in a first manner based on the detected predetermined action.

[0198] The control method of the control system of structure 8 achieves the same effect as the control system of structure 1.

Claims

1. A control system, which is a vehicle control system, comprising: The first processor controls the opening and closing of the vehicle's doors; The second processor authenticates whether an external person present outside the vehicle is a registered user. The second processor performs the following processing: Obtain an exterior image of the vehicle as an external image; Based on the obtained images of the vehicle exterior, detect people or objects outside the vehicle. The system employs a two-tiered authentication process, consisting of a first authentication and a second authentication, to verify whether the person outside the vehicle is a registered user. In the Level 2 certification, Based on each of the vehicle exterior images, a skeletal model similar to the posture of the detected person outside the vehicle is generated, and first proximity mode data is generated by arranging the skeletal models generated based on each of the vehicle exterior images in a time sequence. The time-varying skeletal model shown in the first proximity mode data is compared with the time-varying skeletal model shown in the second proximity mode data pre-stored in the memory. When the memory contains second proximity mode data with a consistency rate of more than a specified value with the first proximity mode data, a second authentication is performed to verify that the detected person outside the vehicle is the registered user. An image representing a portion of the face of the person outside the vehicle, who has been authenticated as the registered user through the second authentication, is extracted from the exterior image. This extracted image is compared with facial data of the registered user pre-recorded in the memory. A first authentication process is then performed to verify whether the person outside the vehicle, who has been authenticated as the registered user through the second authentication, is indeed the registered user. The features of the actions of the person outside the vehicle who is authenticated as the registered user through the Level 2 authentication are extracted from the exterior image. If the action content identified based on the extracted action features is action content pre-recorded in the memory, it is determined that a specified action has been detected. If the detected prescribed action is a pre-recorded action by an external person who is a registered user through the Level 2 authentication, the first processor is sent with first door control content information pre-recorded in the memory corresponding to the detected prescribed action. The first door control content information represents control content related to opening the vehicle door. The first processor controls the opening and closing of the vehicle's doors based on the received first door control content information.

2. The control system according to claim 1, wherein, If multiple people outside the vehicle are detected, including users of the vehicle who are not the registered user (i.e., non-registered users) and multiple people outside the vehicle who are the registered user, the second processor sends second door control content information pre-recorded in the memory to the first processor. The second door control content information represents the control content related to opening the vehicle door when the non-registered user and the registered user are present together.

3. The control system according to claim 2, wherein, If the prescribed action of the unregistered user is detected after the first door control content information is sent to the first processor, the second processor sends the third door control content information, which is pre-recorded in the memory in association with the prescribed action of the unregistered user, to the first processor. The third door control content information represents the control content related to opening the vehicle door and indicates the control content when the prescribed action is performed by the unregistered user. The first door control content information is the door control content information based on the prescribed action of the person outside the vehicle who is authenticated as the registered user through the Level 2 authentication.

4. The control system according to claim 1, wherein, The prescribed actions are gestures specific to the vehicle. If the gesture detected by the second processor is a gesture of covering the face with a hand, a door opening control prohibition message is sent to the first processor. The door opening control prohibition message indicates control content for prohibiting the execution of control to open the vehicle door.

5. The control system according to claim 1, wherein, Upon detecting multiple registered users and the prescribed actions performed by each of the registered users, the second processor sends the first door control content information to the first processor based on the prescribed actions of the registered user located closest to the driver's seat of the vehicle.

6. The control system according to any one of claims 1 to 5, wherein, The second processor performs the following processing: Based on the exterior image of the vehicle, detect the person getting off the vehicle; The prescribed actions of the detected disembarking individuals are detected; The first processor is sent fourth door control content information, which corresponds to the prescribed action of the detected person getting off the vehicle and is pre-stored in the memory. The fourth door control content information represents control content related to closing the vehicle door.

7. A control method for a control system, specifically a control method for a vehicle control system. In the control method of this control system, Obtain an exterior image of the vehicle as an external image; Based on the obtained images of the vehicle exterior, detect people or objects outside the vehicle. The system uses a two-level authentication process, involving both first and second authentication, to verify whether the person outside the vehicle is a registered user. In the Level 2 certification, Based on each of the vehicle exterior images, a skeletal model similar to the posture of the detected person outside the vehicle is generated, and first proximity mode data is generated by arranging the skeletal models generated based on each of the vehicle exterior images in a time sequence. The time-varying skeletal model shown in the first proximity mode data is compared with the time-varying skeletal model shown in the second proximity mode data pre-stored in the memory. When the memory contains second proximity mode data with a consistency rate of more than a specified value with the first proximity mode data, a second authentication is performed to verify that the detected person outside the vehicle is the registered user. An image representing a portion of the face of the person outside the vehicle, who has been authenticated as the registered user through the second authentication, is extracted from the exterior image. This extracted image is compared with facial data of the registered user pre-recorded in the memory. A first authentication process is then performed to verify whether the person outside the vehicle, who has been authenticated as the registered user through the second authentication, is indeed the registered user. The features of the actions of the person outside the vehicle who is authenticated as the registered user through the Level 2 authentication are extracted from the exterior image. If the action content identified based on the extracted action features is action content pre-recorded in the memory, it is determined that a specified action has been detected. If the detected prescribed action is a pre-recorded action by an external person who is a registered user through the Level 2 authentication, the system sends first door control content information, pre-recorded in the memory, corresponding to the detected prescribed action, to the first processor. This first door control content information represents control content related to opening the vehicle's door. The first processor controls the opening and closing of the vehicle's doors based on the received first door control content information.

Images