Information processing device, vehicle control system, information processing method, and program

Abstract

An information processing device (10) is provided with a first authentication unit (13), a first vehicle control unit (14), a movement determination unit (16), and a second vehicle control unit (17). The first authentication unit (13) controls biometric authentication with respect to a person included in a first photographed image generated by photographing a scene around a vehicle, and determines whether or not the person included in the first photographed image is a registered user. When it is determined that the person included in the first photographed image is a registered user, the first vehicle control unit (14) transmits a first control signal for performing first control related to unlocking of an opening and closing body of the vehicle. The movement determination unit (16) controls a movement determination as to whether or not the movement of a person included in a second photographed image generated by photographing a scene around the vehicle or the inside of the vehicle is a registered movement. When the movement of the person included in the second photographed image is a registered movement, the second vehicle control unit (17) transmits a second control signal for performing second control with respect to either a prescribed opening and closing body of the vehicle or the power source of the vehicle, in response to the movement of the person.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a vehicle control system, an information processing method, and a non-transitory computer-readable medium, and more particularly to an information processing device, a vehicle control system, an information processing method, and a non-transitory computer-readable medium for controlling a vehicle. [Background technology]

[0002] Although vehicle doors are unlocked without contact using electronic keys such as smart keys, in recent years, vehicle thefts caused by relay attacks have become serious. Therefore, in order to prevent spoofing, a technology has been proposed for unlocking vehicle doors using electronic keys and biometric authentication. For example, Patent Document 1 discloses a keyless entry device that authenticates an electronic key carried by a person based on information obtained by human body communication, and unlocks the door when it is determined that the electronic key is authentic and that a detected face image based on a captured image matches a registered face image.

[0003] However, the method described in the above-mentioned Patent Document 1 requires an electronic key to unlock the vehicle door, which is inconvenient, and there is also the problem that if the electronic key is stolen, the vehicle door cannot be unlocked.

[0004] Therefore, a technology has been proposed for unlocking the vehicle door using biometric authentication without using an electronic key. For example, a technology for unlocking the vehicle door using biometric authentication and motion has been developed. As an example, Patent Document 2 discloses an unlocking control device that authenticates whether an intruder is a specified user based on image information of an object located within a detection area defined around the vehicle, and unlocks the door based on behavior information acquired based on the image information.

[0005] Technologies have also been developed that use biometric authentication and motion to control the opening and closing of vehicle doors. As an example, Patent Document 3 discloses a vehicle control device that detects registered users from captured images of the vehicle's surroundings based on the gait of people in the vicinity, detects the gaze direction of the registered users from the captured images, and changes the door from a closed state to an open state based on the gaze direction. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2011-032782 A [Patent Document 2] JP 2020-100963 A [Patent Document 3] Patent Publication No. 2021-001447 Summary of the Invention [Problem to be solved by the invention]

[0007] Here, in order to improve convenience, there is a demand for contactless operation of opening and closing doors in addition to unlocking. However, the above-mentioned Patent Document 2 does not disclose such door opening and closing operations, and convenience is insufficient. This is not limited to door opening and closing operations, but also applies to other operations that require the user's intention. Moreover, the above-mentioned Patent Document 3 does not disclose unlocking the vehicle doors without using an electronic key, and convenience is insufficient.

[0008] In view of the above-mentioned problems, an object of the present disclosure is to provide an information processing device, a vehicle control system, an information processing method, and a non-transitory computer-readable medium that improve the convenience of vehicle operation. [Means for solving the problem]

[0009] An information processing device according to an aspect of the present disclosure includes a first authentication means, a first vehicle control means, a motion determination means, and a second vehicle control means. The first authentication means controls biometric authentication of a person included in a first captured image based on a first captured image generated by capturing a scene around a vehicle, and determines whether or not the person included in the first captured image is a registered user. When it is determined that the person included in the first captured image is the registered user, the first vehicle control means transmits a first control signal for executing a first control related to unlocking an opening / closing body of the vehicle. The motion determination means controls motion determination of whether or not a motion of the person included in the second captured image is a registered motion based on a second captured image generated by capturing a scene around the vehicle or an interior of the vehicle. When the motion of the person included in the second captured image is the registered motion, the second vehicle control means transmits a second control signal for executing a second control. The second control is a second control according to a motion of the person on a predetermined opening / closing body of the vehicle or a power source of the vehicle, and is different from the first control.

[0010] A vehicle control system according to an embodiment of the present disclosure includes a biometric authentication device that performs biometric authentication, and an information processing device. The information processing device includes a first authentication means, a first vehicle control means, a motion determination means, and a second vehicle control means. The first authentication means causes the biometric authentication device to perform biometric authentication on a person included in a first captured image based on a first captured image generated by capturing a scene around the vehicle, and determines whether or not the person included in the first captured image is a registered user. When it is determined that the person included in the first captured image is the registered user, the first vehicle control means transmits a first control signal for executing a first control related to unlocking an opening and closing body of the vehicle. The motion determination means controls motion determination of whether or not a motion of the person included in the second captured image is a registered motion based on a second captured image generated by capturing a scene around the vehicle or an interior of the vehicle. When the motion of the person included in the second captured image is the registered motion, the second vehicle control means transmits a second control signal for executing a second control. The second control is a second control according to an action of the person on a predetermined opening / closing object of the vehicle or a power source of the vehicle, and is different from the first control.

[0011] The information processing method according to one aspect of the present disclosure includes a first authentication stage, a first vehicle control stage, a motion determination stage, and a second vehicle control stage. The first authentication stage is a stage of controlling biometric authentication of a person included in a first captured image based on a first captured image generated by capturing a scene around a vehicle, and determining whether or not the person included in the first captured image is a registered user. The first vehicle control stage is a stage of transmitting a first control signal for executing a first control related to unlocking an opening / closing body of the vehicle when it is determined that the person included in the first captured image is the registered user. The motion determination stage is a stage of controlling motion determination of whether or not a motion of a person included in the second captured image is a registered motion based on a second captured image generated by capturing a scene around the vehicle or an interior of the vehicle. The second vehicle control stage is a stage of transmitting a second control signal for executing a second control when a motion of a person included in the second captured image is the registered motion. The second control is a second control according to a motion of the person on a predetermined opening / closing body of the vehicle or a power source of the vehicle, and is different from the first control.

[0012] A non-transitory computer-readable medium according to an embodiment of the present disclosure stores a program for causing a computer to execute a first authentication stage, a first vehicle control stage, an action determination stage, and a second vehicle control stage. The first authentication stage is a stage for controlling biometric authentication of a person included in a first captured image generated by capturing a scene around a vehicle, based on the first captured image, and determining whether or not the person included in the first captured image is a registered user. The first vehicle control stage is a stage for transmitting a first control for executing a first control related to unlocking an opening / closing body of the vehicle, when it is determined that the person included in the first captured image is the registered user. The action determination stage is a stage for controlling action determination of whether or not the action of the person included in the second captured image is a registered action, based on a second captured image generated by capturing a scene around the vehicle or an interior of the vehicle. The second vehicle control stage is a stage for transmitting a second control signal for executing a second control, when the action of the person included in the second captured image is the registered action. The second control is a second control according to the action of the person on a predetermined opening / closing body of the vehicle or a power source of the vehicle, and is different from the first control. Effect of the Invention

[0013] The present disclosure makes it possible to provide an information processing device, a vehicle control system, an information processing method, and a non-transitory computer-readable medium that improve the convenience of vehicle operation. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first embodiment. [Diagram 2] 4 is a flowchart showing the flow of information processing according to the first embodiment. [Diagram 3] FIG. 11 is a block diagram showing the overall configuration of a vehicle control system according to a second embodiment. [Figure 4] FIG. 11 is a block diagram showing the configuration of a face recognition device according to a second embodiment. [Diagram 5]11 is a flowchart showing the flow of face information registration processing according to the second embodiment. [Figure 6] 11 is a flowchart showing the flow of face recognition processing according to the second embodiment. [Figure 7] FIG. 11 is a schematic configuration diagram of an in-vehicle system according to a second embodiment. [Figure 8] FIG. 11 is a block diagram showing a configuration of an in-vehicle system according to a second embodiment. [Figure 9] FIG. 11 is a diagram showing an example of a data structure of user information according to the second embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of a data structure of an action table according to the second embodiment. [Figure 11] 13 is a flowchart showing the flow of a registration process according to the second embodiment. [Figure 12] 10 is a flowchart showing the flow of a control process according to the second embodiment. [Figure 13] FIG. 11 is a sequence diagram showing an example of the flow of a registration process according to the second embodiment. [Figure 14] FIG. 11 is a sequence diagram showing an example of the flow of a control process according to the second embodiment. [Figure 15] FIG. 11 is a sequence diagram showing another example of the flow of the control process according to the second embodiment. [Figure 16] FIG. 11 is a block diagram showing a configuration of an in-vehicle system according to a third embodiment. [Figure 17] FIG. 13 is a diagram showing an example of a user position according to the third embodiment. [Figure 18] FIG. 13 is a diagram showing another example of a user position according to the third embodiment. [Figure 19] FIG. 13 is a diagram illustrating an example of a data structure of an action table according to the third embodiment. [Figure 20] 11 is a flowchart showing an example of the flow of a control process according to a third embodiment. [Figure 21] FIG. 11 is a block diagram showing a configuration of an in-vehicle system according to a fourth embodiment. [Figure 22] 13 is a flowchart showing an example of the flow of a control process according to a fourth embodiment. [Figure 23] FIG. 13 is a block diagram showing a configuration of an in-vehicle system according to a fifth embodiment. [Figure 24] FIG. 13 is a block diagram showing the overall configuration of a vehicle control system according to a sixth embodiment. [Diagram 25] FIG. 23 is a diagram showing an example of a data structure of a motion judgment model table according to the sixth embodiment. [Figure 26] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and duplicated descriptions will be omitted as necessary for clarity of explanation.

[0016] <Issues of the embodiment> Here, the problem of the embodiment will be explained again. In recent years, electronic keys such as smart keys have been used to unlock vehicle doors without contact. However, if the electronic key is stolen, the vehicle doors cannot be unlocked. Recently, there has been a growing problem of vehicle thefts caused by relay attacks.

[0017] Here, the above-mentioned Patent Document 1 discloses a method for unlocking a vehicle door using an electronic key and face authentication. However, as described above, the electronic key is required to unlock the vehicle door, which is problematic in terms of convenience.

[0018] Therefore, a technology has been proposed that uses biometric authentication to unlock vehicle doors without using an electronic key. In addition to unlocking doors, there is also a demand for contactless opening and closing of doors. However, the methods described in the above-mentioned Patent Documents 2 and 3 have difficulty in satisfying such demands.

[0019] Therefore, a method can be considered in which, if the biometric authentication is successful, both the door is unlocked and opened. Although this method can improve convenience from the viewpoint of automatic unlocking, it has a problem that the door is always opened after being unlocked, which may result in unintended door opening.

[0020] It is also possible to use a method in which both the door unlocking and the door opening are performed based on biometric authentication and motion, but this method has the problem that it cannot accommodate cases in which the user only wants to unlock the door. This is not limited to door opening and closing operations, but also applies to other operations that require the user's intention.

[0021] Therefore, there is a strong demand for improving the convenience of vehicle operation. The present embodiment has been made to solve such a problem.

[0022] <Embodiment 1> First, a first embodiment of the present disclosure will be described. FIG. 1 is a block diagram showing a configuration of an information processing device 10 according to the first embodiment. The information processing device 10 is a computer for executing unlocking control of an opening / closing body of a vehicle and various other vehicle-related controls. The information processing device 10 acquires a captured image generated by capturing a scene around the vehicle or a captured image generated by capturing an interior of the vehicle, and executes the above-mentioned control based on the acquired captured image. A captured image generated by capturing a scene around the vehicle under a predetermined condition is called a first captured image. For example, the first captured image is a captured image generated by capturing a scene around the vehicle when the opening / closing body of the vehicle is in a locked state. In addition, a captured image generated by capturing a scene around the vehicle or a vehicle interior under a predetermined condition is called a second captured image. For example, the second captured image is a captured image generated by capturing a scene around the vehicle when the opening / closing body of the vehicle is in an unlocked state. The information processing device 10 includes a first authentication unit 13, a first vehicle control unit 14, an operation determination unit 16, and a second vehicle control unit 17.

[0023] The first authentication unit 13 is also called a first authentication means. The first authentication unit 13 controls biometric authentication for a person included in the first captured image based on the first captured image, and determines whether or not the person included in the first captured image is a registered user. In addition, controlling the biometric authentication may mean that the first authentication unit 13 executes the biometric authentication, or that the first authentication unit 13 causes another device, for example, a biometric authentication device, to execute the biometric authentication. Here, the biometric authentication is biometric authentication capable of extracting biometric information from a captured image, such as face authentication, iris authentication, vein authentication, and ear authentication.

[0024] The first vehicle control unit 14 is also called a first vehicle control means. When the first vehicle control unit 14 determines that the person included in the first captured image is a registered user, the first vehicle control unit 14 transmits a first control signal for executing a first control. The first control is a control related to unlocking an opening and closing body of the vehicle.

[0025] The motion determination unit 16 is also called motion determination means. The motion determination unit 16 controls motion determination as to whether or not the motion of the person included in the second captured image is a registered motion, based on the second captured image. In addition, controlling the motion determination may mean that the motion determination unit 16 executes the motion determination, or that the motion determination unit 16 causes another device, for example, a motion determination device, to execute the motion determination.

[0026] The second vehicle control unit 17 is also called a second vehicle control means. When the movement of the person included in the second captured image is a registered movement, the second vehicle control unit 17 transmits a second control signal for executing a second control regarding the vehicle. Here, the second control is a control for a predetermined opening / closing body of the vehicle or a power source of the vehicle, and is a control different from the first control. The second control is determined according to the movement of the person. The predetermined opening / closing body of the vehicle may be the same as the opening / closing body of the vehicle controlled by the first control, may be a part of the opening / closing body of the vehicle controlled by the first control, or may be another opening / closing body included in the vehicle.

[0027] FIG. 2 is a flowchart showing a flow of information processing according to the first embodiment. First, the first authentication unit 13 of the information processing device 10 judges whether or not the captured image acquired by the information processing device 10 is the first captured image generated by capturing a scene around the vehicle (S11). If the captured image acquired by the information processing device 10 is the first captured image (Yes in S11), the first authentication unit 13 controls biometric authentication for a person included in the first captured image based on the first captured image (S12). Next, the first authentication unit 13 judges whether or not the person included in the first captured image is a registered user based on the result of the biometric authentication (S13). If it is determined that the person included in the first captured image is a registered user (Yes in S13), the first vehicle control unit 14 transmits a first control signal for unlocking the opening and closing body of the vehicle (S14), and the process returns to step S11. On the other hand, if it is not determined that the person included in the first captured image is a registered user (No in S13), the first authentication unit 13 returns the process to step S11 as it is.

[0028] On the other hand, if the captured image acquired by the information processing device 10 is not the first captured image (No in S11), the motion determination unit 16 determines whether or not it is a second captured image generated by capturing a landscape around the vehicle or the interior of the vehicle (S15). If the captured image acquired by the information processing device 10 is not the second captured image (No in S15), the motion determination unit 16 returns the process to step S11. On the other hand, if the captured image is the second captured image (Yes in S15), the motion determination unit 16 controls the motion determination of the person included in the second captured image based on the second captured image (S16). Then, if the motion of the person included in the second captured image is a registered motion (Yes in S17), the second vehicle control unit 17 transmits a second control signal for executing a second control corresponding to the motion (S18), and returns the process to step S11. On the other hand, if the motion of the person included in the second captured image is not a registered motion (No in S17), the motion determination unit 16 returns the process to step S11.

[0029] As described above, according to the first embodiment, the information processing device 10 executes control of unlocking the opening and closing mechanism of the vehicle by biometric authentication, and executes control other than unlocking the vehicle by motion determination. This allows the user to unlock the vehicle door and perform other vehicle-related operations in a non-contact manner without using an electronic key.

[0030] In addition, since the information processing device 10 performs control other than unlocking the vehicle based on the user's intention by determining the motion, it is possible to prevent the door from being opened unintentionally, compared to a method in which the door is unlocked and opened when the biometric authentication is successful. Also, compared to a method in which the door is unlocked and opened by two-factor authentication of biometric authentication and motion, the information processing device 10 can handle cases in which the user only wants to unlock the door. Therefore, the convenience of vehicle operation is improved.

[0031] <Embodiment 2> Next, a second embodiment of the present disclosure will be described. FIG. 3 is a block diagram showing an overall configuration of a vehicle control system 900 according to the second embodiment. The vehicle control system 900 is a computer system for implementing various controls related to a vehicle using biometric authentication. The various controls related to a vehicle include controls related to unlocking or locking an opening / closing body of the vehicle, and other controls that require the user's will. The opening / closing body may be a front door, a rear door, a bonnet, a fuel filler cover, a door glass (window), a trunk cover, and the like. Here, the control related to unlocking an opening / closing body of the vehicle is called a first control, the control related to locking an opening / closing body of the vehicle is called a third control, and other controls that require the user's will are called a second control. The second control may be, for example, a control related to opening or closing an opening / closing body of the vehicle, or may be starting or stopping the power supply of the vehicle. The power supply of the vehicle may be an ignition power supply (IG power supply) or an accessory power supply (ACC power supply).

[0032] In the second embodiment, the biometric authentication is face authentication, but may be other biometric authentication capable of extracting biometric information from a captured image, such as iris authentication, vein authentication, or ear authentication.

[0033] The vehicle control system 900 includes in-vehicle systems 1-1, 1-2, and 1-3 and a face authentication device 2, which are communicatively connected via a network N. Here, the network N is a wired or wireless communication line.

[0034] The in-vehicle systems 1-1, 1-2, and 1-3 are mounted on different vehicles, and each executes a first control, a second control, and a third control of the corresponding vehicle. Hereinafter, when the in-vehicle systems 1-1, 1-2, and 1-3 are not to be distinguished from one another, they may be simply referred to as the in-vehicle systems 1. Note that the number of in-vehicle systems 1 included in the vehicle control system 900 is not limited to three.

[0035] The in-vehicle system 1 controls face authentication and executes the first control based on the face authentication result. In the second embodiment, face authentication is performed by the face authentication device 2. Therefore, the in-vehicle system 1 controls face authentication by transmitting a face authentication request for the user U who is the authentication target to the face authentication device 2 and receiving the face authentication result from the face authentication device 2. The face authentication request includes a face image which is a photographed image including at least the face area of ​​the user U, but may include face information extracted from the face image instead of the face image. The in-vehicle system 1 also determines the behavior of the user U who is the authentication target, and executes the second control based on the behavior determination result. The in-vehicle system 1 also executes the third control under a predetermined condition.

[0036] The face authentication device 2 is an example of a biometric authentication device that performs biometric authentication. The face authentication device 2 is an information processing device that stores face information of multiple people. In response to a face authentication request received from the outside, the face authentication device 2 compares the face image or face information included in the request with the face information of each user U, and returns the comparison result (face authentication result) to the request source.

[0037] 4 is a block diagram showing the configuration of a face authentication device according to embodiment 2. The face authentication device 2 includes a face information database (DB) 210, a detection unit 220, a feature point extraction unit 230, a registration unit 240, and an authentication unit 250. The face information DB 210 stores a user ID 211 and face feature information 212 of the user ID in association with each other. The face feature information 212 is a collection of feature points extracted from a face image, and is an example of face information.

[0038] The detection unit 220 detects a face area included in a registration image for registering face information, and supplies the face area to the feature point extraction unit 230. The feature point extraction unit 230 extracts feature points from the face area detected by the detection unit 220, and supplies the face feature information to the registration unit 240. The feature point extraction unit 230 also extracts feature points included in a face image for authentication, and supplies the face feature information to the authentication unit 250.

[0039] The registration unit 240 issues a new user ID 211 when registering the facial feature information. The registration unit 240 associates the issued user ID 211 with the facial feature information 212 extracted from the registered image and registers them in the facial information DB 210. The authentication unit 250 performs face authentication using the facial feature information 212. Specifically, the authentication unit 250 compares the facial feature information extracted from the face image with the facial feature information 212 in the facial information DB 210. The authentication unit 250 returns a match of the facial feature information to the information processing device 100. The match of the facial feature information corresponds to the success or failure of authentication. The match of the facial feature information (matching) refers to a case where the degree of match is equal to or greater than a predetermined value.

[0040] FIG. 5 is a flowchart showing the flow of the face information registration process according to the second embodiment. First, the face authentication device 2 acquires a registration image of the user U included in the face information registration request (S21). For example, the face authentication device 2 accepts the face information registration request from the information processing device 100 or a user terminal (not shown) via the network N. Next, the detection unit 220 detects a face area included in the registration image (S22). Next, the feature point extraction unit 230 extracts feature points from the face area detected in step S22 and supplies face feature information to the registration unit 240 (S23). Finally, the registration unit 240 issues a user ID 211, associates the user ID 211 with face feature information 212, and registers the user ID 211 and face feature information 212 in the face information DB 210 (S24). Note that the face authentication device 2 may receive the face feature information 212 from the source of the face information registration request, and register the face feature information 212 in the face information DB 210 in association with the user ID 211.

[0041] FIG. 6 is a flowchart showing the flow of the face authentication process according to the second embodiment. First, the feature point extraction unit 230 acquires face feature information for authentication (S31). For example, the face authentication device 2 receives a face authentication request from the information processing device 100 via the network N, and extracts face feature information from the captured image included in the face authentication request as in steps S21 to S23. Alternatively, the face authentication device 2 may receive face feature information from the information processing device 100. Next, the authentication unit 250 collates the acquired face feature information with the face feature information 212 in the face information DB 210 (S32). If the face feature information matches, that is, if the degree of match of the face feature information is equal to or greater than a predetermined value (Yes in S33), the authentication unit 250 specifies the user ID 211 of the user whose face feature information matches (S34). Then, the authentication unit 250 returns the fact that face authentication was successful and the specified user ID 211 as a face authentication result to the information processing device 100 (S35). If there is no matching facial feature information (No in S33), the authentication unit 250 returns a message indicating that the facial authentication has failed as a facial authentication result to the information processing device 100 (S36).

[0042] 7 is a schematic configuration diagram of an in-vehicle system 1 according to a second embodiment. The in-vehicle system 1 is mounted on a vehicle V. The in-vehicle system 1 includes at least an information processing device 100 and an in-vehicle camera mounted on the vehicle V. The in-vehicle camera is connected to the information processing device 100. In the second embodiment, there are multiple in-vehicle cameras, and the multiple in-vehicle cameras include a peripheral camera 120 that captures the surrounding scenery outside the vehicle V and an in-vehicle camera 122 that captures the interior of the vehicle V.

[0043] First, when a user U wants to unlock and get into a locked vehicle V, the user U approaches the vicinity of the vehicle V. The surrounding camera 120 then captures an image of the user U located in the vicinity of the vehicle V to generate a first captured image. The information processing device 100 then controls face recognition based on the first captured image and determines whether or not the user U is a predefined registered user. If the information processing device 100 determines that the user U is a registered user, it controls the door to be unlocked (first control). Note that face recognition for unlocking the door may be referred to as first face recognition.

[0044] When the door is unlocked, the user U performs a predetermined action (gesture). At this time, the peripheral camera 120 captures the user U located in the vicinity of the vehicle V and generates a second captured image. In the second embodiment, the second captured image is an image different from the first captured image described above, but may be the same image. Then, the information processing device 100 controls the action determination of whether or not the action of the user U is a registered action registered in advance based on the second captured image. When the information processing device 100 determines that the action of the user U is a registered action, it executes a second control according to the action of the user U. Note that the target and type of the second control may be determined according to the registered action. For example, when the user U performs an action of changing the direction of the face from right to left, the information processing device 100 may open the door of the passenger seat, and when the user U performs an action of changing the direction of the line of sight from bottom to top, the information processing device 100 may start the IG power supply.

[0045] The second captured image may be captured when the user U is in the vehicle. For example, when the door is unlocked, the user U opens the driver's door and gets into the driver's seat. The user U then performs a predetermined action. At this time, the in-vehicle camera 122 captures the user U located in the vehicle V and generates a second captured image. The information processing device 100 then controls the action determination as to whether the action of the user U is a registered action that has been registered in advance, based on the second captured image. When the information processing device 100 determines that the action of the user U is a registered action, it executes the second control according to the action of the user U. For example, when the user U performs an action of changing the direction of his / her line of sight from bottom to top, the information processing device 100 may turn on the IG power supply of the vehicle V.

[0046] Furthermore, when the user U gets off the vehicle and the doors are closed, the information processing device 100 performs control to lock the doors when a predetermined time has elapsed since the doors were closed (third control).

[0047] 8 is a block diagram showing the configuration of an in-vehicle system 1 according to embodiment 2. The in-vehicle system 1 includes an information processing device 100, a peripheral camera 120, an in-vehicle camera 122, a first driving unit 130, and second driving units 140-1 and 140-5.

[0048] The information processing device 100 is an example of the above-mentioned information processing device 10. The information processing device 100 includes a registration unit 101, an acquisition unit 102, a first authentication unit 103, a first vehicle control unit 104, an operation determination unit 106, a second vehicle control unit 107, and a storage unit 108.

[0049] The registration unit 101 is also called a registration means. The registration unit 101 transmits a face information registration request including a registration image to the face authentication device 2 via the network N. The registration unit 101 also stores a user ID issued by the face authentication device 2 when registering the face information as user information in the storage unit 108. The user ID recorded as the user information is the user ID of a registered user of the vehicle V.

[0050] 9 is a diagram showing an example of a data structure of user information according to embodiment 2. As shown in this figure, the user information includes a user ID for each registered ID.

[0051] Returning to Fig. 8, the description will continue. The registration unit 101 registers an action for executing the second control in the action table. The registered action is called a registered action. 10 is a diagram illustrating an example of a data structure of an action table according to embodiment 2. The action table associates the contents of a registered action with the contents of a second control.

[0052] The content of the registered action is information that associates the action target with the action type, and is identified by the action ID. The action target is the body or a part of the body that is to be moved, and in this figure, it is the person's face or gaze. However, it is not limited to this, and the action target may be a hand. The action type is information that indicates how to move the action target. The action type may be "changing the direction" such as "changing the direction from right to left" or "changing the direction from up to down". The action type may also be "(the body) approaching the vehicle V". The action type may also be "touching the face area or a specific area around the face" such as "touching the ear (with the hand)" or "touching the nose (with the hand)". Note that the content of one registered action may be composed of one set of action target and action type, or may be composed of multiple sets of action target and action type. In other words, the content of the registered action is determined based on at least one of the direction of the person's face and the direction of the gaze, and may be, for example, "changing the direction of the face from right to left to front to down to up". The registration action may also be a person "touching the ears, nose, and mouth in that order."

[0053] The content of the second control is information associating a second control object with a second control type. The second control object is a part of the vehicle V that is to be controlled by the second control, and in this figure, it is a door or an IG power source. However, the second control object is not limited to this, and may be other opening and closing objects such as a door glass, a bonnet, a fuel filler cover, or a trunk cover, or may be other power sources such as an ACC power source. The second control type is information indicating how to control the second control object. The second control type may be control related to opening and closing, such as "opening (the opening and closing object)" or "closing (the opening and closing object)", or control related to starting and stopping the power source, such as "starting (the power source)" or "stopping (the power source)". In other words, the content of the second control may be door opening, door closing, starting the IG power source, stopping the IG power source, starting the ACC power source, stopping the ACC power source, opening the fuel filler cover, closing the fuel filler cover, opening the trunk cover, closing the trunk cover, opening the bonnet, closing the bonnet, opening the door glass (window), or closing the door glass. The content of the second control in 1 may be composed of one set of second control objects and second control types, or may be composed of multiple sets of second control objects and second control types. In other words, the content of the second control is determined based on at least one of opening or closing of the opening / closing body and starting or stopping of the power source, and may be, for example, "opening the door and starting the IG power source."

[0054] The direction of control indicated by the second control type may be determined in advance to correspond to the direction of change indicated by the action type. In other words, when the action target is a face or line of sight, the content of the second control may indicate control for opening and closing the opening / closing body of the vehicle V in a direction according to the line of sight or face direction indicated by the action type. For example, when the content of the registered action is "changing the face direction from right to left," the content of the second control may be "sliding the sliding door from right to left to open it." This allows the user U to perform intuitive actions, improving operability.

[0055] Returning to FIG. 8, the description will be continued. The acquisition unit 102 is also called an acquisition means. The acquisition unit 102 is connected to the peripheral camera 120 and the in-vehicle camera 122, and acquires captured images from the peripheral camera 120 and the in-vehicle camera 122. When the door of the vehicle V is locked, the acquisition unit 102 supplies the captured image acquired from the peripheral camera 120 to the first authentication unit 103 as a first captured image. When the door of the vehicle V is unlocked, the acquisition unit 102 supplies the captured image acquired from the peripheral camera 120 or the in-vehicle camera 122 to the operation determination unit 106 as a second captured image. Note that the second captured image is an image captured at a different timing from the first captured image, but when the door of the vehicle V is unlocked by a first control signal described later, the acquisition unit 102 may supply the same image as the first captured image to the operation determination unit 106 as the second captured image.

[0056] The acquisition unit 102 may continuously acquire images from the peripheral camera 120 and the in-vehicle camera 122 at predetermined time intervals. In this case, when the door of the vehicle V is locked and a predetermined first captured image acquisition condition is satisfied, the acquisition unit 102 may supply the captured image acquired from the peripheral camera 120 as the first captured image to the first authentication unit 103. The first captured image acquisition condition may also be called a first authentication start condition. The first captured image acquisition condition may be that the user U included in the captured image captured by the peripheral camera 120 is located in a predetermined area (first captured image acquisition area) based on the vehicle V. For example, whether the user U is located in the first captured image acquisition area may be determined based on whether the size, width, or length of the face area of ​​the user U included in the captured image captured by the peripheral camera 120 is equal to or greater than a predetermined number of pixels. Also, for example, whether the user U is located in the first captured image acquisition area may be determined based on whether the length connecting the feature points of a predetermined facial organ included in the face area of ​​the user U included in the captured image captured by the peripheral camera 120 is equal to or greater than a predetermined number of pixels. The feature points of the facial organs may be at least two of the right eye, the left eye, the end of the nose, the end of the mouth, the end of the right eyebrow, and the end of the left eyebrow. As an example, the feature points of the facial organs are the right eye and the left eye. For example, whether the user U is located in the first captured image acquisition area may be determined based on the position of the person area by detecting a person area from an overhead image obtained by combining images captured by a plurality of vehicle-mounted cameras mounted on the vehicle V. For example, whether the user U is located in the first captured image acquisition area may be determined based on whether a sensor such as a distance measuring sensor detects the presence of a person in the first captured image acquisition area. In this way, by not starting the process by the first authentication unit 103 until the first captured image acquisition condition is satisfied, power consumption can be reduced.

[0057] In addition, when the door of the vehicle V is unlocked and a predetermined second photographed image acquisition condition is satisfied, the acquisition unit 102 may supply the photographed image acquired from the peripheral camera 120 or the in-vehicle camera 122 as the second photographed image to the motion determination unit 106. The second photographed image acquisition condition may also be called a motion determination start condition. The second photographed image acquisition condition may be that the user U included in the photographed image captured by the peripheral camera 120 or the in-vehicle camera 122 is located in a predetermined area (second photographed image acquisition area) based on the vehicle V. The second photographed image acquisition area may be an area different from the first photographed image acquisition area, or may be the same area. The determination of whether the user U is located in the second photographed image acquisition area will be omitted by replacing the "peripheral camera 120" in the above description with the "peripheral camera 120 or the in-vehicle camera 122" and the "first photographed image acquisition area" with the "second photographed image acquisition area."

[0058] In addition, when the user U is located in the second photographed image acquisition area, the acquisition unit 102 may immediately determine that the second photographed image acquisition condition is satisfied and supply the photographed image to the motion determination unit 106 as the second photographed image. However, instead of this, the acquisition unit 102 may supply the photographed image to the motion determination unit 106 as the second photographed image after determining that the user U stays in the second photographed image acquisition area for a predetermined time or more. In other words, the second photographed image acquisition condition may be that the user U included in the photographed image stays in the second photographed image acquisition area for a predetermined time or more. In this case, for example, the second photographed image acquisition condition may be that the user U included in the photographed image stops in the second photographed image acquisition area for several seconds or more, or sits in the driver's seat for several seconds or more. In this way, by not starting the motion determination until the second photographed image acquisition condition is satisfied, power consumption can be reduced.

[0059] The first authentication unit 103 is an example of the above-mentioned first authentication unit 13. The first authentication unit 103 causes the face authentication device 2 to execute face authentication for a person (user U) included in the first captured image, and determines whether or not the person included in the first captured image is a registered user based on the face authentication result. Specifically, the first authentication unit 103 transmits a face authentication request including the first captured image to the face authentication device 2 via the network N. Then, the first authentication unit 103 receives the face authentication result from the face authentication device 2 via the network N. Then, the first authentication unit 103 determines whether or not a user ID included in the face authentication result is included in the user information stored in the storage unit 108. If the user ID included in the face authentication result is included in the user information, the first authentication unit 103 determines that the person included in the first captured image is a registered user. Then, the first authentication unit 103 supplies the registered user determination result to the first vehicle control unit 104. Note that the process by the first authentication unit 103 may be referred to as a first authentication process.

[0060] The first vehicle control unit 104 is an example of the above-mentioned first vehicle control unit 14. The first vehicle control unit 104 is connected to the first drive unit 130. When the first vehicle control unit 104 determines that the person included in the first captured image is a registered user, the first vehicle control unit 104 transmits a first control signal for executing a first control related to unlocking the opening and closing body of the vehicle V to the first drive unit 130.

[0061] Furthermore, when the opening / closing body of the vehicle V is unlocked and closed and a predetermined time has passed since it was closed, the first vehicle control unit 104 transmits a third control signal for executing a third control related to locking the opening / closing body of the vehicle V to the first drive unit 130. As a result, the opening / closing body of the vehicle V is automatically locked without using an electronic key, improving the security level and convenience.

[0062] The motion determination unit 106 is an example of the motion determination unit 16 described above. The motion determination unit 106 acquires the second captured image from the acquisition unit 102. Then, the motion determination unit 106 executes motion determination as to whether or not the motion of the person (user U) included in the second captured image is a registered motion based on the second captured image. Specifically, the motion determination unit 106 generates motion information for authentication indicating a feature amount of the motion from the second captured image. Then, the motion determination unit 106 refers to a motion table stored in the storage unit 108, and determines whether or not the motion information for authentication matches motion information for registration, which is a feature amount of the registered motion registered in the motion table. The motion information for authentication matches the motion information for registration may mean that the similarity of the feature amount is equal to or greater than a predetermined threshold. The motion determination unit 106 executes a series of processes from the generation of motion information for authentication to motion determination using a motion determination model. The motion determination model is a determiner that is learned in correspondence with the registered motion registered in the motion table. For example, the motion determination model may receive the second captured image as input and output a similarity between the authentication motion information and the registration motion information. In this case, the motion determination model may include a convolutional neural network (CNN). When the authentication motion information matches the registration motion information, the motion determination unit 106 notifies the second vehicle control unit 107 of the type of motion.

[0063] The second vehicle control unit 107 is an example of the above-mentioned second vehicle control unit 17. The second vehicle control unit 107 is connected to the second driving unit 140-1 and the second driving unit 140-5. When the motion of the user U included in the second captured image is a registered motion, the second vehicle control unit 107 transmits a second control signal and executes a second control determined according to the motion of the user U. The destination of the second control signal is either the second driving unit 140-1 or the second driving unit 140-5, and is determined according to the motion of the user U.

[0064] The storage unit 108 is a storage device that stores information necessary for information processing, that is, registration processing and control processing, of the information processing device 100. The storage unit 108 stores at least user information, an action table, and an action determination model corresponding to the registered actions.

[0065] The first driving unit 130 is an actuator that generates a driving force for unlocking and locking the opening / closing body of the vehicle V. The second driving unit 140-1 is an actuator that generates a driving force for opening and closing the opening / closing body of the vehicle V. The second driving unit 140-5 is a switch for starting and stopping the power supply of the vehicle V.

[0066] FIG. 11 is a flowchart showing the flow of the registration process according to the second embodiment. First, the registration unit 101 of the information processing device 100 acquires a registration image of the user U (S40). At this time, the registration unit 101 may acquire the registration image from the peripheral camera 120 or the in-car camera 122, or may acquire the registration image from a user terminal (not shown) used by the user U via the network N. Next, the registration unit 101 requests the face authentication device 2 to register face information in the face information DB 210 via the network N (S41). The face information registration request includes the registration image. Next, the registration unit 101 acquires a user ID from the face authentication device 2 via the network N (S42). Next, the registration unit 101 registers the user ID as user information (S43).

[0067] Next, the registration unit 101 acquires the operation information for registration (S44) and registers the operation information for registration in the operation table (S45). The operation information for registration includes the content of the second control and the content of the registration operation. The registration unit 101 may acquire the content of the second control and the content of the registration operation by input from the user U of the vehicle V via the input unit. Specifically, the user U first inputs the second control object and the second control type, and sets the content of the second control. For example, the user U may set the content of the second control by selecting one or more sets of the second control object and the second control type from a list of the second control object and the second control type that are predetermined. Then, the user U inputs the operation object and the operation type, and sets the content of the registration operation corresponding to the content of the set second control. For example, the user U may set the content of the registration operation by selecting one set of the operation object and the operation type from a list of the operation object and the operation type that is predetermined. The user U may also set the content of the registration operation by selecting multiple sets of the operation object and the operation type from a list of the operation object and the operation type that is predetermined, and freely combining them. The registration unit 101 then registers the registration action information, specifically, information associating the content of the registration action set based on the input from the user U via the input unit with the content of the second control, in the action table. The registration unit 101 may specify the content of the registration action based on video data of a registered user performing the registration action, and register the specified content of the registration action in the action table in association with the content of the second control. If the action table is set in advance, the processes shown in steps S44 to S45 may be omitted.

[0068] FIG. 12 is a flowchart showing the flow of the control process according to the second embodiment. First, the information processing device 100 determines whether the door of the vehicle V is locked (S50). Here, the information processing device 100 may determine whether the door of the vehicle V is locked by, for example, monitoring the state of the first driving unit 130 in the first vehicle control unit 104. If the door of the vehicle V is locked (Yes in S50), the acquisition unit 102 determines whether the above-mentioned first captured image acquisition condition is satisfied (S51). If the first captured image acquisition condition is not satisfied (No in S51), the acquisition unit 102 returns the process to step S50. On the other hand, if the first captured image acquisition condition is satisfied (Yes in S51), the acquisition unit 102 supplies the captured image acquired from the peripheral camera 120 to the first authentication unit 103 as the first captured image, and the process proceeds to step S52. In step S52, the first authentication unit 103 transmits a face authentication request to the face authentication device 2 via the network N. At this time, the first authentication unit 103 includes at least one of the first captured image, the face area extracted from the first captured image, and the face feature information extracted from the face area in the face authentication request. Then, the first authentication unit 103 receives the face authentication result from the face authentication device 2 via the network N (S53). If the face authentication is successful, the face authentication result includes a message to that effect and a user ID, and if the face authentication is unsuccessful, the face authentication result includes a message to that effect.

[0069] Next, the first authentication unit 103 judges whether or not the face authentication is successful (S54). If the face authentication is not successful (No in S54), the first authentication unit 103 returns the process to step S50. On the other hand, if the face authentication is successful (Yes in S54), the first authentication unit 103 identifies the user ID whose face authentication is successful (S56) and proceeds to step S57. Specifically, the first authentication unit 103 extracts the user ID included in the face authentication result. In step S57, the first authentication unit 103 judges whether or not the identified user ID is included in the user information stored in the storage unit 108, thereby judging whether or not the user U whose face authentication is successful is a registered user (S57). If the user U whose face authentication is successful is not a registered user (No in S57), the first authentication unit 103 returns the process to step S50. On the other hand, if the user U whose face has been successfully authenticated is a registered user (Yes in S57), the first vehicle control unit 104 sends a first control signal to the first drive unit 130 to unlock the door of the vehicle V (S58).

[0070] On the other hand, if the door of the vehicle V is not locked, that is, unlocked (No in S50), the acquisition unit 102 determines whether or not the above-mentioned second captured image acquisition condition is satisfied (S59). If the second captured image acquisition condition is not satisfied (No in S59), the acquisition unit 102 proceeds to step S63. On the other hand, if the second captured image acquisition condition is satisfied (Yes in S59), the acquisition unit 102 supplies the captured image acquired from the peripheral camera 120 or the in-vehicle camera 122 to the motion determination unit 106 as the second captured image, and proceeds to step S60. In step S60, the motion determination unit 106 detects a person area or a face area included in the second captured image, and generates motion information for authentication of the user U included in the second captured image based on the person area or the face area (S60). Next, the motion determination unit 106 determines whether the motion of the user U is a registered motion by determining whether the motion information for authentication of the user U matches the motion information for registration of the registered motion registered in the motion table (S61). If the motion of the user U is not a registered motion (No in S61), the motion determination unit 106 advances the process to step S63. On the other hand, if the motion of the user U is a registered motion (Yes in S61), the motion determination unit 106 advances the process to step S62. In step S62, the second vehicle control unit 107 generates a second control signal corresponding to the motion of the user U (i.e., the matching registered motion) and transmits the second control signal to the second driving unit 140 (S62). Specifically, the second vehicle control unit 107 identifies the second control object and the second control type corresponding to the registered motion using the motion table, and generates the second control signal based on the second control object and the second control type. Then, the second vehicle control unit 107 transmits the second control signal to the second driving unit 140 corresponding to the second control object. Then, the second vehicle control unit 107 advances the process to step S63.

[0071] In step S63, the information processing device 100 determines whether the door of the vehicle V is in a closed state. Here, the information processing device 100 may determine whether the door of the vehicle V is in a closed state by, for example, monitoring the state of the second drive unit 140-1 in the second vehicle control unit 107. If the door of the vehicle V is not in a closed state, that is, in an open state (No in S63), the information processing device 100 returns the process to step S50. On the other hand, if the door of the vehicle V is in a closed state (Yes in S63), the information processing device 100 determines whether a predetermined time Δt has elapsed since the door of the vehicle V became in a closed state (S64). If the predetermined time Δt has not elapsed (No in S64), the process returns to step S50. On the other hand, if the predetermined time Δt has elapsed (Yes in S64), the first vehicle control unit 104 transmits a third control signal for locking the door of the vehicle V to the first drive unit 130 (S65), and returns the process to step S50.

[0072] FIG. 13 is a sequence diagram showing an example of the flow of the registration process according to the second embodiment. First, the camera photographs the user U (S501) and transmits the photographed image to the information processing device 100 as a registration image (S502). The camera photographing the user U may be the peripheral camera 120 or the in-car camera 122. Alternatively, a camera (not shown) of a user terminal may photograph the user U, and the user terminal may transmit the photographed image to the information processing device 100 via the network N. Next, the information processing device 100 transmits a face information registration request including the registration image to the face authentication device 2 via the network N (S503). Then, the face authentication device 2 registers face information (facial feature information) of the user U based on the registration image included in the received face information registration request (S504). Then, the face authentication device 2 notifies the information processing device 100 of the user ID via the network N (S505). The information processing device 100 registers the notified user ID as user information (S506). Then, the information processing device 100 accepts an input operation of the motion information for registration from the registered user (S507). Then, the information processing device 100 registers the accepted motion information for registration, specifically, information associating the content of the registration motion with the content of the second control, in the motion table (S508).

[0073] 14 is a sequence diagram showing an example of the flow of the control process according to the embodiment 2. This diagram shows the control process when a user U gets into a vehicle V.

[0074] First, the peripheral camera 120 photographs the user U (S521) and transmits the photographed image to the information processing device 100 via the network N (S522). Here, it is assumed that the first photographed image acquisition condition is satisfied. In this case, the photographed image received by the information processing device 100 is identified as the first photographed image. Next, the information processing device 100 transmits a face authentication request for the face area of ​​the user U in the received photographed image to the face authentication device 2 via the network N (S523). Then, the face authentication device 2 performs face authentication on the face area of ​​the user U in the first photographed image included in the received face authentication request (S524). Here, it is assumed that the face authentication is successful. The face authentication device 2 transmits a face authentication result including a notice of successful face authentication and a user ID to the information processing device 100 via the network N (S525).

[0075] The information processing device 100, which has received the face authentication result, compares the user ID included in the face authentication result with the user information to determine whether the user U included in the first captured image is a registered user (S526). Next, the information processing device 100 transmits a first control signal for unlocking the door of the vehicle V to the first driving unit 130 as the first control (S527). As a result, the door of the vehicle V becomes unlocked.

[0076] Next, the peripheral camera 120 captures the user U (S528) and transmits the captured image to the information processing device 100 via the network N (S529). Here, it is assumed that the second captured image acquisition condition is satisfied. In this case, the captured image received by the information processing device 100 is identified as the second captured image. The motion determination unit 106 of the information processing device 100 executes motion determination as to whether or not the motion of the user U included in the second captured image is a registered motion based on the second captured image (S530). Here, it is assumed that the user U performs a motion of moving the direction of his / her face from right to left. In this case, the information processing device 100 refers to the motion table and determines that the motion of the user U matches the motion ID "G1". Then, the information processing device 100 transmits a second control signal to the second driving unit 140-1 as the second control to open the door of the vehicle V (S531). As a result, the door of the vehicle V is in an open state.

[0077] In step S530, the information processing device 100 executes the motion determination based on the second captured image. However, instead of this, the information processing device 100 may specify the first captured image or a frame of a predetermined period including the capture timing of the first captured image as the second captured image, and execute the motion determination based on the second captured image. In this case, steps S528 to S529 may be omitted.

[0078] FIG. 15 is a sequence diagram showing another example of the flow of the control process according to the second embodiment. This diagram also shows the control process when the user U gets into the vehicle V. First, the same process as steps S521 to S527 in FIG. 14 is executed. Next, it is assumed that the user U opens the door of the vehicle V and gets into the vehicle V. That is, the user U moves from the outside of the vehicle V to the inside of the vehicle V. Here, the in-car camera 122 captures the user U (S532) and transmits the captured image to the information processing device 100 via the network N (S533). Here, it is assumed that the second captured image acquisition condition is satisfied. In this case, the captured image received by the information processing device 100 is identified as the second captured image. Then, based on the second captured image, the information processing device 100 executes a motion determination as to whether or not the motion of the user U included in the second captured image is a registered motion (S534). Here, it is assumed that the user U moves the direction of his / her gaze from bottom to top. In this case, the information processing device 100 refers to the action table and determines that the action of the user U matches the action ID "G3". Then, the information processing device 100 transmits a second control signal to the second drive unit 140-5 as the second control to start the IG power supply of the vehicle V (S535). As a result, the IG power supply of the vehicle V becomes active.

[0079] Furthermore, the process from step S532 onwards may be performed after step S531 in FIG.

[0080] Thus, according to the second embodiment, the information processing device 100 executes control of unlocking the doors of the vehicle V by biometric authentication, and executes control other than unlocking the vehicle V, which requires the will of the user U, by motion determination. This allows the user U to unlock the vehicle doors without using an electronic key, and to perform other operations related to the vehicle as intended. The user U can also execute only unlocking. Therefore, the same effects as those of the first embodiment can be achieved, and the convenience of operating the vehicle is improved.

[0081] <Embodiment 3> Next, a third embodiment of the present disclosure will be described. The third embodiment is characterized in that the content of the registered motion that is the subject of motion judgment or the content of the second control can differ depending on the user's position.

[0082] 16 is a block diagram showing a configuration of an in-vehicle system 1a according to embodiment 3. The in-vehicle system 1a differs from the in-vehicle system 1 in that the in-vehicle system 1a includes peripheral cameras 120-1 to 120-5, second driving units 140-1 to 140-6, and an information processing device 100a, instead of the peripheral camera 120, the second driving units 140-1 and 140-5, and the information processing device 100. Note that the number of peripheral cameras 120 and second driving units 140 is not limited to this.

[0083] The peripheral cameras 120-1 to 120-5 are peripheral cameras 120 mounted at different positions on the vehicle V. For example, the peripheral cameras 120-1, 120-2, 120-3, 120-4, and 120-5 are cameras that capture the scenery on the left front, left side, right front, right side, and rear of the vehicle V, respectively. The peripheral cameras 120-1 to 120-5 are connected to an acquisition unit 102a, which will be described later.

[0084] The second drive units 140-1, 140-2, 140-3, and 140-4 are actuators that generate drive forces for opening and closing various opening / closing bodies of the vehicle V. As an example, the second drive units 140-1, 140-2, 140-3, and 140-4 are actuators that generate drive forces for opening and closing the doors, the hood, the fuel filler cover, and the trunk cover of the vehicle V, respectively.

[0085] The second drive units 140-5 and 140-6 are switches for starting and stopping various power sources of the vehicle V. As an example, the second drive units 140-5 and 140-6 are switches for starting and stopping the IG power source and the ACC power source of the vehicle V, respectively. The second driving units 140-1 to 140-6 are connected to a second vehicle control unit 107a, which will be described later.

[0086] The information processing device 100a includes, instead of the acquisition unit 102, the operation determination unit 106, and the second vehicle control unit 107, an acquisition unit 102a, an operation determination unit 106a, and a second vehicle control unit 107a.

[0087] The acquiring unit 102a has basically the same functions as the acquiring unit 102. However, the acquiring unit 102a differs from the acquiring unit 102 in that the acquiring unit 102a identifies a second captured image from captured images acquired from the peripheral cameras 120-1 to 120-5 and the in-vehicle camera 122, and identifies a user position that is an area in which the user U is located and is included in the second captured image. The user position may be identified based on the result of "determination of whether or not the user U is located in the second captured image acquisition area" in the determination of the second captured image acquisition condition described in the above-mentioned second embodiment.

[0088] Here, the second captured image acquisition area that is the second captured image acquisition condition may be different for each camera. FIG. 17 is a diagram showing an example of a user position according to the third embodiment. The second captured image acquisition areas of the peripheral cameras 120-1, 120-2, 120-3, 120-4, and 120-5 are areas A1, A2, B1, B2, and C1, respectively. The second captured image acquisition area of ​​the in-vehicle camera 122 is area D1. The areas A1, A2, B1, B2, C1, and D1 are determined in advance based on the vehicle V. When it is determined that the user U is located in any of the areas A1, A2, B1, B2, C1, and D1, the acquisition unit 102a specifies the second captured image and supplies the second captured image and the user position to the operation determination unit 106a.

[0089] For example, the acquisition unit 102a specifies as the second captured image a captured image in which the size, width, or length of the face area of ​​the user U is equal to or greater than a predetermined number of pixels among the captured images acquired from the peripheral cameras 120-1 to 120-5 and the in-vehicle camera 122. Also, for example, the acquisition unit 102a specifies as the second captured image a captured image in which the length connecting the feature points of a predetermined facial organ included in the face area of ​​the user U is equal to or greater than a predetermined number of pixels. Then, the acquisition unit 102a specifies the camera that captured the second captured image, and specifies the second captured image acquisition area corresponding to the camera as the area where the user is located (user position). In this case, the acquisition unit 102a specifies that the camera that captured the second captured image is the peripheral camera 120-4, and specifies that the user position is area B2.

[0090] FIG. 18 is a diagram showing another example of a user position according to the third embodiment. The second captured image acquisition area is area T 1 and region D1. Region T 1 includes areas A1, A2, B1, B2, and C1 corresponding to the peripheral cameras 120-1, 120-2, 120-3, 120-4, and 120-5, respectively. The areas A1, A2, B1, B2, C1, and D1 in this figure are also determined in advance with the vehicle V as a reference.

[0091] For example, the acquisition unit 102a synthesizes the captured images acquired from the peripheral cameras 120-1, 120-2, 120-3, 120-4, and 120-5 to generate an overhead image of the vehicle V viewed from above. The acquisition unit 102a then detects a person area from the overhead image and determines whether the person area is area T 1 If it is located within the region T 1The acquisition unit 102a identifies an area including the position of the person area within the area as the user position. Alternatively, an omnidirectional camera may be mounted at a predetermined position of the vehicle V, and the acquisition unit 102a may use an image generated by the omnidirectional camera as the overhead image. The acquisition unit 102a then identifies a camera corresponding to the user position, and identifies the captured image acquired from the camera as the second captured image. The acquisition unit 102a then supplies the second captured image and the user position to the motion determination unit 106a. In this case, the acquisition unit 102a identifies the user position as area B2, and identifies the captured image captured by the peripheral camera 120-4 corresponding to area B2 as the second captured image.

[0092] In the example of Figure 18, if the captured image acquired from the in-vehicle camera 122 satisfies the second captured image acquisition condition, the acquisition unit 102a may identify the second captured image and the user position in a manner similar to that shown in Figure 17, and supply them to the operation determination unit 106a.

[0093] Returning to Fig. 16, the description will be continued. The motion determination unit 106a has basically the same functions as the motion determination unit 106, but controls motion determination according to the user position using a motion table. More specifically, the motion determination unit 106a uses the motion table to determine whether or not the motion of the user U included in the second captured image is a registered motion corresponding to the area in which the user U is located.

[0094] The second vehicle control unit 107a has basically the same functions as the second vehicle control unit 107. However, the second vehicle control unit 107a differs from the second vehicle control unit 107 in that the second vehicle control unit 107a transmits a second control signal for executing a second control corresponding to each movement of a person included in the second captured image and an area in which the person is located.

[0095] FIG. 19 is a diagram showing an example of a data structure of the action table according to the third embodiment. The contents of the registered actions in the action table are determined corresponding to each predetermined area based on the vehicle V. For example, when the user position is in the area C1, the contents of the registered actions are action IDs "G41" and "G42". That is, the action determination unit 106a determines whether the action of the user U corresponds to the action IDs "G41" and "G42". On the other hand, when the user position is in the area D1, the contents of the registered actions are action IDs "G11", "G12", "G21", and "G22". That is, the action determination unit 106a determines whether the action of the user U corresponds to the action IDs "G11", "G12", "G21", and "G22".

[0096] The content of the second control in the action table, that is, the second control object or the second control object type, is also determined corresponding to each predetermined area based on the vehicle V. For example, when the user position is in area A1 and the user U performs an action with action ID "G11", the second vehicle control unit 107a transmits a second control signal for executing the second control with the content "open the door". On the other hand, even when the user U similarly performs an action with action ID "G11", if the user position is in area D1, the second vehicle control unit 107a transmits a second control signal indicating the second control with the content "activate the ACC power supply".

[0097] 20 is a flowchart showing an example of the flow of the control process according to embodiment 3. The steps shown in FIG. 20 include steps S70 to S73 instead of steps S60 to S62 shown in FIG.

[0098] If the second captured image acquisition condition is satisfied in step S59, the acquisition unit 102a identifies the camera that captured the second captured image, and identifies the second captured image acquisition area corresponding to the camera as the user position (S70). The acquisition unit 102a then supplies the information on the second captured image and the user position to the motion determination unit 106a. Next, the motion determination unit 106a detects a person area or a face area included in the second captured image, and generates motion information for authentication of the user U included in the second captured image based on the person area or the face area (S71). Next, the motion determination unit 106a determines whether the motion information for authentication of the user U matches the motion information for registration of the registration motion corresponding to the user position registered in the motion table, and determines whether the motion of the user U is the registration motion corresponding to the user position (S72). If the motion of the user U is not the registration motion corresponding to the user position (No in S72), the motion determination unit 106a advances the process to step S63. On the other hand, if the motion of the user U is a registered motion corresponding to the user position (Yes in S72), the motion determination unit 106a advances the process to step S73. In step S73, the second vehicle control unit 107a generates a second control signal corresponding to the motion of the user U (i.e., the matching registered motion) and the user position, and transmits the second control signal to the second driving unit 140 (S73). Specifically, the second vehicle control unit 107 a uses the operation table to identify a second control object and a second control type according to the registered operation and the user position, and generates a second control signal based on the second control object and the second control type. a transmits a second control signal to second drive unit 140 corresponding to the second control object. Then, second vehicle control unit 107a advances the process to step S63.

[0099] In this way, in the third embodiment, the user U can use different actions for operation depending on the position. The user U can also change the content of the second control for each area in which the user U is located. This improves convenience.

[0100] <Embodiment 4> Next, a fourth embodiment of the present disclosure will be described. The fourth embodiment is characterized in that the information processing device starts a motion determination when it determines that a person included in the second captured image is a registered user by using biometric authentication. Note that, in the fourth embodiment, the biometric authentication is face authentication, but is not limited thereto.

[0101] 21 is a block diagram showing a configuration of an in-vehicle system 1b according to the fourth embodiment. The in-vehicle system 1b has basically the same configuration and functions as the in-vehicle system 1 according to the second embodiment. However, the in-vehicle system 1b includes an information processing device 100b instead of the information processing device 100. The information processing device 100b basically has the same configuration as the information processing device 100, but includes a second authentication unit 105 and an operation determination unit 106b instead of the operation determination unit 106.

[0102] The second authentication unit 105 is also called a second authentication means. Based on the second captured image provided from the acquisition unit 102, the second authentication unit 105 controls face authentication for a person (user U) included in the second captured image, and judges whether or not the user U included in the second captured image is a registered user. The control of face authentication for the user U included in the second captured image by the second authentication unit 105 is similar to the control of face authentication for the user U included in the first captured image by the first authentication unit 103. For example, the second authentication unit 105 causes the face authentication device 2 to execute face authentication for the user U included in the second captured image, and judges whether or not the person included in the second captured image is a registered user based on the face authentication result. Then, the second authentication unit 105 provides the registered user judgment result to the operation judgment unit 106b. The process by the second authentication unit 105 may be called a second authentication process.

[0103] The motion determining section 106b basically has the same functions as the motion determining section 106. However, the motion determining section 106b executes motion determination when it is determined that the person included in the second captured image is a registered user.

[0104] 22 is a flowchart showing an example of the flow of the control process according to embodiment 4. The steps shown in FIG. 22 include steps S80 to S81 in addition to the steps shown in FIG.

[0105] If the second captured image acquisition condition is satisfied in step S59, the acquisition unit 102 supplies the captured image acquired from the peripheral camera 120 or the in-vehicle camera 122 to the second authentication unit 105 as the second captured image, and the process proceeds to step S80. In step S80, the second authentication unit 105 executes the same process as that shown in steps S52 to S56 for the second captured image. Then, the second authentication unit 105 determines whether the user U whose face authentication has been successful is a registered user by determining whether the user information stored in the storage unit 108 includes the specified user ID (S81). If the user U whose face authentication has been successful is not a registered user (No in S81), the second authentication unit 105 proceeds to step S63. On the other hand, if the user U whose face authentication has been successful is a registered user (Yes in S81), the movement determination unit 106b executes the process shown in step S60 and generates the movement information for authentication of the user U included in the second captured image. Step S61 and subsequent steps are the same as those in FIG.

[0106] As described above, according to the fourth embodiment, the information processing device 100b does not start the motion determination when the biometric authentication based on the second captured image fails, and therefore it is possible to avoid executing the second control in response to the motion of a person other than the registered user. This makes it possible to enhance security. Also, it is possible to reduce power consumption.

[0107] <Embodiment 5> Next, a fifth embodiment of the present disclosure will be described. Fig. 23 is a block diagram showing a configuration of an in-vehicle system 1c according to the fifth embodiment. The in-vehicle system 1c has basically the same configuration and functions as the in-vehicle system 1b according to the fourth embodiment. However, the in-vehicle system 1c includes a light 150, a speaker 152, and an information processing device 100c instead of the information processing device 100b.

[0108] The light 150 functions as a light output unit that outputs light. The speaker 152 functions as an audio output unit that outputs sound. The sound may be a warning sound or an operation sound. The light 150 and the speaker 152 are an example of an alarm unit that is mounted on the vehicle V and notifies the user U of specific information. The light 150 and the speaker 152 may be a light and a speaker that are standard equipment on the vehicle V. The alarm unit is connected to the alarm control unit 109 and is controlled by the alarm control unit 109. The alarm unit may include a display unit instead of or in addition to the light 150.

[0109] The information processing device 100c basically includes a notification control unit 109 in addition to the configuration of the information processing device 100b.

[0110] The notification control unit 109 is also called a notification control means. The notification control unit 109 uses the light 150 and the speaker 152 to notify the start or result of various processes by the information processing device 100c. The various processes may be at least one of the first authentication process by the first authentication unit 103, the second authentication process by the second authentication unit 105, or the operation determination process by the operation determination unit 106b. The notification control unit 109 may control the light 150 and the speaker 152 so that the notification mode differs depending on the type of process or whether the process has started or ended.

[0111] For example, when the first authentication process is started, the notification control unit 109 blinks the light 150 once. The first authentication process may be started when the first captured image acquisition condition is satisfied. When the first authentication process is completed and it is determined that the user U is a registered user, the notification control unit 109 outputs an operation sound once from the speaker 152. When the second authentication process is started, the notification control unit 109 blinks the light 150 twice. When the second authentication process is started, the notification control unit 109 outputs an operation sound twice from the speaker 152. When the second authentication process is completed and it is determined that the user U is a registered user, the notification control unit 109 blinks the light 150 three times. When the motion determination process is completed and it is determined that the motion of the user U is a registered motion, the notification control unit 109 outputs an operation sound three times from the speaker 152.

[0112] As described above, according to the fifth embodiment, the information processing device 100c notifies the user U of the start or results of various processes, so that the user U can easily grasp the state of the vehicle V. For example, when the control of the vehicle V is unintentionally executed, the user U can quickly respond.

[0113] <Embodiment 6> Next, a sixth embodiment of the present disclosure will be described. The sixth embodiment is a modification of the fifth embodiment, and is characterized in that the information processing device uses a motion determination model learned for each registered user in the motion determination process.

[0114] 24 is a block diagram showing an overall configuration of a vehicle control system 900d according to embodiment 6. The vehicle control system 900d includes a management server 3 in addition to the configuration of the vehicle control system 900.

[0115] The management server 3 acquires captured images of registered users performing registered actions from each in-vehicle system 1 via the network N, and learns a motion determination model for each user. In this way, the management server 3 generates a motion determination model that reflects the motion habits of each user. The management server 3 registers the generated motion determination model in a motion determination model table T in association with the user ID. The motion determination model table T may be stored in a storage unit (not shown) of the management server 3. Then, in response to a request from the in-vehicle system 1, the management server 3 transmits the motion determination model associated with the user ID in the motion determination model table T to the in-vehicle system 1.

[0116] Furthermore, the management server 3 may register the learned parameters of the motion determination model in the motion determination model table T in association with a user ID, and transmit the parameters of the motion determination model associated with the user ID to the in-vehicle system 1 in response to a request from the in-vehicle system 1.

[0117] The information processing device 100c of the in-vehicle system 1 transmits a request for generating a motion determination model to the management server 3 via the network N during the initial registration process. The request for generating a motion determination model includes the user ID of the registered user notified from the face authentication device 2 and a captured image of the registered user performing the registration operation. The information processing device 100c of the in-vehicle system 1 receives a motion determination model corresponding to the registered user from the management server 3 via the network N. When the registered user of the vehicle V is changed and a motion determination model corresponding to the changed registered user has already been generated, the information processing device 100c of the in-vehicle system 1 transmits a request for acquiring the motion determination model to the management server 3 via the network N. The request for acquiring the motion determination model includes the user ID of the changed registered user. The information processing device 100c of the in-vehicle system 1 receives a motion determination model corresponding to the changed registered user from the management server 3 via the network N.

[0118] Then, the information processing device 100c of the in-vehicle system 1 stores the motion judgment model received from the management server 3 in the storage unit 108 in association with the user ID of the registered user. When the second authentication unit 105 determines that the user U is a registered user, the motion judgment unit 106b executes motion judgment using the motion judgment model corresponding to the user ID of the registered user.

[0119] 25 is a diagram showing an example of the data structure of the movement determination model table T according to embodiment 6. In this figure, the movement determination model table T associates a user ID with a movement determination model ID for identifying a movement determination model.

[0120] Thus, according to the sixth embodiment, the information processing device 100c controls the motion judgment using the motion judgment model learned for each registered user, and therefore the motion judgment accuracy can be improved. Also, the motion judgment model corresponding to the user ID is stored in the management server 3, and the motion judgment model is supplied to the in-vehicle system 1 as necessary. Therefore, even when registered users are frequently replaced in rental cars or car sharing, or when the user U replaces the vehicle V, the motion judgment model corresponding to the user U can be easily introduced into the in-vehicle system 1. Therefore, convenience is improved.

[0121] In the sixth embodiment, the management server 3 learns a motion determination model for each registered user, but the motion determination model may be learned for each user attribute. As an example, the user attribute may be gender or age. In this case, the management server 3 may associate the user attribute with the motion determination model and register it in the motion determination model table T. Furthermore, the registration unit 101 of the information processing device 100c of the in-vehicle system 1 may register information in which the user ID and the user attribute information are associated as user information during the registration process. The user attribute information may be acquired by the registration unit 101 via an input unit (not shown) that accepts input from the user, or may be acquired by the registration unit 101 from a user terminal via the network N. During the initial registration process or when the registered user is changed, the information processing device 100c may receive a motion determination model according to the user attribute corresponding to the user ID of the registered user from the management server 3, and store it in the storage unit 108 in association with the user ID.

[0122] Next, a description will be given of the physical configuration of the face authentication device 2, the information processing devices 100, 100a, 100b, and 100c, and the management server 3. Fig. 26 is a diagram showing a configuration example of a computer that can be used as the face authentication device 2, the information processing devices 100, 100a, 100b, and 100c, and the management server 3. The computer 1000 has a processor 1010, a storage unit 1020, a ROM 1030, a RAM 1040, a communication interface (IF: Interface) 1050, and a user interface 1060.

[0123] The communication interface 1050 is an interface for connecting the computer 1000 to a communication network via a wired communication means or a wireless communication means, etc. The user interface 1060 includes a display unit such as a display, and also includes an input unit such as a keyboard, a mouse, and a touch panel.

[0124] The storage unit 1020 is an auxiliary storage device that can hold various data. The storage unit 1020 does not necessarily need to be a part of the computer 1000, and may be an external storage device or a cloud storage connected to the computer 1000 via a network.

[0125] The ROM 1030 is a non-volatile storage device. The programs executed by the processor 1010 may be stored in the storage unit 1020 or the ROM 1030. The storage unit 1020 or the ROM 1030 stores various programs for implementing the functions of each unit in the face authentication device 2, the information processing devices 100, 100a, 100b, and 100c, and the management server 3, for example.

[0126] The above program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium includes Random-Access Memory (RAM), Read-Only Memory (ROM), Flash memory, Solid-State Drive (SSD) or other memory technology, CD-ROM, Digital Versatile Disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

[0127] The RAM 1040 is a volatile storage device. Various semiconductor memory devices such as a dynamic random access memory (DRAM) or a static random access memory (SRAM) are used for the RAM 1040. The RAM 1040 can be used as an internal buffer for temporarily storing data and the like. The processor 1010 loads a program stored in the storage unit 1020 or the ROM 1030 into the RAM 1040 and executes it. The processor 1010 may be a central processing unit (CPU) or a graphics processing unit (GPU). The processor 1010 executes a program to realize the functions of each unit in the face recognition device 2, the information processing devices 100, 100a, 100b, 100c, and the management server 3. The processor 1010 may have an internal buffer for temporarily storing data and the like.

[0128] In the above-mentioned embodiment, the computer is configured as a computer system including a personal computer, a word processor, etc. However, the present invention is not limited to this, and the computer can also be configured as a server of a LAN (local area network), a host for computer (personal computer) communication, a computer system connected to the Internet, etc. It is also possible to distribute functions to each device on the network and configure a computer as a whole network.

[0129] The present disclosure is not limited to the above-mentioned embodiments, and can be appropriately modified without departing from the spirit of the present disclosure. For example, the above-mentioned embodiments 2 to 6 can be arbitrarily combined. For example, the embodiment 2 can be combined with the embodiment 5, and the embodiment 3 can be combined with the embodiment 4 or the embodiment 5.

[0130] In the above-described first embodiment, the second captured image is generated by capturing a landscape around the vehicle or the interior of the vehicle under a predetermined condition, for example, when the opening / closing body of the vehicle is in an unlocked state. In the above-described second to sixth embodiments, when the door of the vehicle V is in an unlocked state, the acquisition unit 102 supplies the captured image acquired from the surrounding camera 120 or the in-vehicle camera 122 and satisfying the second captured image acquisition condition to the operation determination unit 106 as the second captured image. However, the second captured image may also include a captured image generated by capturing an interior of the vehicle when the opening / closing body of the vehicle is in a locked state. Therefore, even when the door of the vehicle V is in a locked state, the acquisition unit 102 may supply the captured image acquired from the in-vehicle camera 122 and satisfying the second captured image acquisition condition to the operation determination unit 106 as the second captured image.

[0131] In addition, in the above-mentioned embodiments 2 to 6, the face recognition device 2 is connected to the information processing device via the network N, but the functions of the face recognition device 2 may be included in the information processing device and connected to various components of the information processing device via an internal bus.

[0132] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) a first authentication means for controlling biometric authentication of a person included in a first captured image generated by capturing an image of a scene around a vehicle, and determining whether or not the person included in the first captured image is a registered user; a first vehicle control means for transmitting a first control signal for executing a first control related to unlocking an opening / closing mechanism of the vehicle when the person included in the first captured image is determined to be the registered user; a motion determination means for controlling a motion determination as to whether or not a motion of a person included in a second captured image is a registered motion, based on a second captured image generated by capturing an image of a scenery around the vehicle or an interior of the vehicle; a second vehicle control means for transmitting a second control signal for executing a second control, which is different from the first control, on a predetermined opening / closing body of the vehicle or a power source of the vehicle in response to the motion of the person, when the motion of the person included in the second captured image is the registered motion; An information processing device comprising: (Appendix 2) The operation determination means controls the operation determination based on the second captured image when the opening / closing mechanism of the vehicle is in an unlocked state. 2. An information processing device according to claim 1. (Appendix 3) The information processing device according to claim 1 or 2, wherein the registration action is determined based on at least one of a direction of a person's gaze and a direction of a person's face. (Appendix 4) The information processing device described in Appendix 3, wherein the second control signal is a control signal for opening and closing the specified opening / closing body of the vehicle in a direction corresponding to the direction of the line of sight or the direction of the face. (Appendix 5) The information processing device according to any one of claims 1 to 4, further comprising a notification control means for notifying the start or result of a process by the first authentication means or a process by the operation determination means. (Appendix 6) The information processing device according to claim 5, wherein the notification control means notifies the start or result using a speaker or a light mounted on the vehicle. (Appendix 7) a second authentication means for controlling biometric authentication of a person included in the second captured image based on the second captured image and determining whether or not the person included in the second captured image is the registered user; The motion determination means controls the motion determination when it is determined that the person included in the second captured image is the registered user. 7. An information processing device according to any one of claims 1 to 6. (Appendix 8) The motion determination means controls the motion determination when a person included in the second captured image is located in a predetermined area based on the vehicle. 8. An information processing device according to any one of claims 1 to 7. (Appendix 9) The movement determination means controls the movement determination when a size or length of a face area of ​​a person included in the second captured image is equal to or greater than a predetermined number of pixels, or when a length connecting predetermined face organs included in a face area of ​​a person included in the second captured image is equal to or greater than a predetermined number of pixels. 9. An information processing device according to any one of claims 1 to 8. (Appendix 10) The registration operation is determined corresponding to each of predetermined areas based on the vehicle, The motion determination means determines whether or not a motion of a person included in the second captured image is a registered motion corresponding to an area in which the person is located. 10. An information processing device according to any one of claims 1 to 9. (Appendix 11) the second control is determined in response to each motion of a person and each predetermined area based on the vehicle; The second vehicle control means transmits a second control signal for executing a second control corresponding to a movement of a person included in the second captured image and an area in which the person is located. 11. An information processing device according to any one of claims 1 to 10. (Appendix 12) A registration means for registering the registration operation based on an input from the registered user is provided. 12. An information processing device according to any one of claims 1 to 11. (Appendix 13) The information processing device according to any one of claims 1 to 12, wherein the motion determination means controls the motion determination using a motion determination model learned for each registered user. (Appendix 14) a biometric authentication device for performing biometric authentication; Information processing device Equipped with The information processing device includes: a first authentication means for causing the biometric authentication device to perform biometric authentication on a person included in a first captured image generated by capturing an image of a scene around a vehicle, and determining whether or not the person included in the first captured image is a registered user; a first vehicle control means for transmitting a first control signal for executing a first control related to unlocking an opening / closing mechanism of the vehicle when the person included in the first captured image is determined to be the registered user; a motion determination means for controlling a motion determination as to whether or not a motion of a person included in a second captured image is a registered motion, based on a second captured image generated by capturing an image of a scenery around the vehicle or an interior of the vehicle; a second vehicle control means for transmitting a second control signal for executing a second control, which is different from the first control, on a predetermined opening / closing body of the vehicle or a power source of the vehicle in response to the motion of the person, when the motion of the person included in the second captured image is the registered motion; A vehicle control system having the above configuration. (Appendix 15) The information processing device includes: a second authentication means for controlling biometric authentication of a person included in the second captured image based on the second captured image and determining whether or not the person included in the second captured image is the registered user; The motion determination means controls the motion determination when it is determined that the person included in the second captured image is the registered user. 15. A vehicle control system as described in claim 14. (Appendix 16) The system further includes a management server for storing the action determination model learned for each registered user, The motion determination means controls the motion determination using a motion determination model corresponding to the registered user acquired from the management server. 16. A vehicle control system as described in claim 15. (Appendix 17) a first authentication stage for controlling biometric authentication of a person included in a first captured image generated by capturing an image of a scene around a vehicle, and determining whether or not the person included in the first captured image is a registered user; a first vehicle control step of transmitting a first control signal for executing a first control related to unlocking an opening and closing mechanism of the vehicle when the person included in the first captured image is determined to be the registered user; a motion determination step of controlling a motion determination as to whether or not a motion of a person included in a second captured image is a registered motion based on a second captured image generated by capturing a scene around the vehicle or an interior of the vehicle; a second vehicle control step of transmitting a second control signal for executing a second control, which is different from the first control, on a predetermined opening / closing body of the vehicle or a power source of the vehicle in response to the person's movement, when the movement of the person included in the second captured image is the registered movement; An information processing method comprising: (Appendix 18) On the computer, a first authentication stage for controlling biometric authentication of a person included in a first captured image generated by capturing an image of a scene around a vehicle, and determining whether or not the person included in the first captured image is a registered user; a first vehicle control step of transmitting a first control signal for executing a first control related to unlocking an opening and closing mechanism of the vehicle when the person included in the first captured image is determined to be the registered user; a motion determination step of controlling a motion determination as to whether or not a motion of a person included in a second captured image is a registered motion based on a second captured image generated by capturing a scene around the vehicle or an interior of the vehicle; a second vehicle control step of transmitting a second control signal for executing a second control, which is different from the first control, on a predetermined opening / closing body of the vehicle or a power source of the vehicle in response to the person's movement, when the movement of the person included in the second captured image is the registered movement; A non-transitory computer-readable medium having stored thereon a program for executing the program. [Explanation of symbols]

[0133] 1,1a,1b,1c In-vehicle systems 2. Face recognition device 3 Management Server 10, 100, 100a, 100b, 100c Information processing system (information processing device) 13,103 First Certification Section 14,104 First vehicle control unit 16,106,106a,106b Operation judgment section 17, 107, 107a Second vehicle control unit 101 Registration Department 102,102a Acquisition part 105 Second Certification Section 108 Storage section 109 Notification control section 120 Peripheral Camera 122 In-car camera 130 First Drive Unit 140 Second Drive Unit 150 Light 152 Speaker 210 Face Information DB 211 User ID 212 Facial Feature Information 220 Detection unit 230 Feature Point Extraction Unit 240 Registration Department 250 Authentication Department 900,900d Vehicle Control System 1000 Computers 1010 Processor 1020 Storage section 1030 ROM 1040 RAM 1050 Communication Interface 1060 User Interface V Vehicle U User N Network Area A B area T Operation Judgment Model Table

Claims

1. A first authentication means for controlling biometric authentication of a person included in a first captured image generated by photographing the person when a vehicle door is locked, and for determining whether the person included in the first captured image is a registered user; a first vehicle control means for transmitting a first control signal for executing a first control related to unlocking an opening / closing body of the vehicle when the person included in the first captured image is determined to be the registered user; a motion determination means for controlling a motion determination as to whether or not a motion of a person included in a second captured image generated by capturing the second captured image when the opening / closing body is in an unlocked state is a registered motion; a second vehicle control means for transmitting a second control signal for executing a second control, which is different from the first control, on a predetermined opening / closing body of the vehicle or a power source of the vehicle in response to the motion of the person, when the motion of the person included in the second captured image is the registered motion; An information processing device comprising:

2. The vehicle further includes a determination means for determining whether an image generated by photographing the scenery around the vehicle or the interior of the vehicle is the second photographed image, When it is determined that an image generated by photographing a scenery around the vehicle or an interior of the vehicle is the second photographed image, the motion determination means controls the motion determination based on the second photographed image. The information processing device according to claim 1 .

3. The information processing apparatus according to claim 1 , wherein the registration action is determined based on at least one of a line of sight direction and a face direction of a person.

4. The information processing device according to claim 3 , wherein the second control signal is a control signal for opening and closing the predetermined opening / closing body of the vehicle in a direction according to the direction of the line of sight or the direction of the face.

5. The information processing apparatus according to claim 1 , further comprising: a notification control unit configured to notify a start or a result of a process by the first authentication unit or a process by the operation determination unit.

6. The information processing device according to claim 5 , wherein the notification control means notifies the user of the start or the result by using a speaker or a light mounted on the vehicle.

7. a second authentication means for controlling biometric authentication of a person included in the second captured image based on the second captured image and determining whether or not the person included in the second captured image is the registered user; The motion determination means controls the motion determination when it is determined that the person included in the second captured image is the registered user. The information processing device according to claim 1 .

8. a biometric authentication device for performing biometric authentication; Information processing device Equipped with The information processing device includes: a first authentication means for causing the biometric authentication device to perform biometric authentication on a person included in a first captured image generated by capturing an image when an opening / closing mechanism of a vehicle is in a locked state, and determining whether or not the person included in the first captured image is a registered user; a first vehicle control means for transmitting a first control signal for executing a first control related to unlocking an opening / closing body of the vehicle when the person included in the first captured image is determined to be the registered user; a motion determination means for controlling a motion determination as to whether or not a motion of a person included in a second captured image generated by capturing the second captured image when the opening / closing body is in an unlocked state is a registered motion; a second vehicle control means for transmitting a second control signal for executing a second control, which is different from the first control, on a predetermined opening / closing body of the vehicle or a power source of the vehicle in response to the motion of the person, when the motion of the person included in the second captured image is the registered motion; A vehicle control system having the above configuration.

9. A first authentication stage for controlling biometric authentication of a person included in a first captured image generated by photographing the person when a vehicle door is locked, and determining whether the person included in the first captured image is a registered user; a first vehicle control step of transmitting a first control signal for executing a first control related to unlocking an opening and closing mechanism of the vehicle when the person included in the first captured image is determined to be the registered user; a motion determination step of controlling a motion determination as to whether a motion of a person included in a second captured image generated by capturing the second captured image when the opening / closing body is in an unlocked state is a registered motion; a second vehicle control step of transmitting a second control signal for executing a second control, which is a second control corresponding to the person's action and different from the first control, on a predetermined opening / closing body of the vehicle or a power source of the vehicle, when the action of the person included in the second captured image is the registered action; An information processing method comprising:

10. On the computer, a first authentication stage for controlling biometric authentication of a person included in a first captured image generated by capturing an image when an opening / closing mechanism of the vehicle is in a locked state, and determining whether or not the person included in the first captured image is a registered user; a first vehicle control step of transmitting a first control signal for executing a first control related to unlocking an opening and closing mechanism of the vehicle when the person included in the first captured image is determined to be the registered user; a motion determination step of controlling a motion determination as to whether a motion of a person included in a second captured image generated by capturing the second captured image when the opening / closing body is in an unlocked state is a registered motion; a second vehicle control step of transmitting a second control signal for executing a second control, which is a second control corresponding to the person's action and different from the first control, on a predetermined opening / closing body of the vehicle or a power source of the vehicle, when the action of the person included in the second captured image is the registered action; A program for executing the above.

11. The motion determination means controls the motion determination using a motion determination model associated with the registered user. The information processing device according to claim 1 .

Images