Vehicle operation detection device and vehicle operation detection method

Abstract

To provide a vehicle operation detection apparatus capable of detecting a user's gesture with improved accuracy.SOLUTION: An operation detection apparatus 200 includes a storage unit 210 that stores a learned model machine-learned using teacher data in association with an image captured in advance and a toe of a user, an approach determination unit 220 that determines whether the toe of the user has approached into a recognition area A3 set in a capturing area A2 of a camera 80 based on a position of the toe of the user in the image obtained by inputting a newly captured image into the learned model, and a gesture determination unit 230 that calculates a displacement vector of the toe of the user based on a plurality of images captured at intervals of time after the toe of the user is determined to have approached into the recognition area A3 and determines whether a gesture is performed or not, according to whether a direction of the displacement vector of the toe of the user is a direction corresponding to the gesture.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a vehicle operation detection device and a vehicle operation detection method. [Background technology]

[0002] Patent Document 1 describes a vehicle that includes a vehicle body with an opening, a vehicle gate that opens and closes the opening, a gate actuator that drives the vehicle gate, a camera that captures images of the vehicle's surroundings, and a vehicle operation detection device that controls the gate actuator. When the vehicle operation detection device determines that a user has performed a predetermined gesture based on an image captured by the camera, it causes the gate actuator to open the vehicle gate. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-196798 A Summary of the Invention [Problem to be solved by the invention]

[0004] The above-described vehicle operation detection device has room for improvement in terms of detecting a user's gesture with high accuracy. [Means for solving the problem]

[0005] The means for solving the above problems and their effects will be described below. The vehicle operation detection device that solves the above problem is applied to a vehicle that includes a vehicle body having an opening, an opening / closing body that opens and closes the opening, a drive unit that operates the opening / closing body to open and close, and a camera that captures images of the periphery of the opening, and detects a user's gesture as a trigger for starting the opening / closing operation of the opening / closing body, and includes a memory unit that stores a trained model that has been machine-learned using teacher data that associates the image captured in advance with a body part used in the user's gesture, an entry determination unit that determines whether or not the body part has entered a recognition area that is set within the shooting area of ​​the camera, based on the position of the body part in the image obtained by inputting a newly captured image into the trained model, and a displacement vector of the body part is calculated based on a plurality of images captured at a time interval after it is determined that the body part has entered the recognition area, and a direction of the displacement vector is determined based on the position of the body part in the image. Did the gesture pattern change in line with the settings? and a gesture determination unit that determines whether or not the gesture has been performed depending on whether or not the gesture has been performed.

[0006] When a body part of a user enters a recognition area, the vehicle operation detection device having the above configuration determines whether or not a gesture using the body part has been performed. Here, the vehicle operation detection device determines whether or not a body part of a user has entered the recognition area using an image captured by a camera and a trained model. Therefore, it is possible to accurately determine that a body part of a user has entered the recognition area. In this way, the vehicle operation detection device can improve the detection accuracy of a user's gesture.

[0007] In the above-mentioned vehicle operation detection device, when, among the multiple images taken at intervals, an image in which the position of the body part can be identified is defined as a specified image, and an image in which the position of the body part cannot be identified is defined as an unidentified image, it is preferable that the gesture determination unit identifies the position of the body part in the unidentified image by matching features of an area including the position of the body part in the specified image with the unidentified image, and calculates the displacement vector based on the position of the body part in the specified image and the unidentified image.

[0008] The vehicle operation detection device can reduce the calculation load when calculating the displacement vector. In the above-mentioned vehicle operation detection device, it is preferable that the gesture determination unit calculates the displacement vector based on the positions of the body parts for each of the multiple images obtained by inputting the multiple images taken at intervals into the trained model.

[0009] The vehicle operation detection device can accurately calculate the displacement vector of a body part. In the above vehicle operation detection device, it is preferable that the gesture determination unit stops determining the gesture when a magnitude of the displacement vector is less than a lower limit determination value.

[0010] For example, when the user does not perform a gesture, such as when the user is stopped in the recognition area, the amount of movement of the user's body part per unit time tends to be "0" or close to "0". In this regard, the vehicle operation detection device configured as described above stops judging the gesture when the magnitude of the displacement vector of the body part is less than the lower limit judgment value. Therefore, the vehicle operation detection device can reduce the risk of erroneously judging the user's gesture.

[0011] In the above-mentioned vehicle operation detection device, it is preferable that the gesture determination unit does not stop judging the gesture even if the magnitude of the displacement vector is less than the lower limit judgment value, if the number of times that the magnitude of the displacement vector becomes less than the lower limit judgment value is less than the number of stop judgments.

[0012] Depending on the state when the user performs a gesture, the displacement speed of the user's body part may temporarily decrease, and the movement amount of the body part per unit time may temporarily become "0" or a value close to "0". In addition, when the user performs a gesture in which the direction of the displacement vector of the body part is reversed, the movement amount of the body part per unit time may temporarily become "0" or a value close to "0". In this regard, the vehicle operation detection device having the above configuration does not stop the gesture determination when the number of times that the magnitude of the displacement vector of the body part becomes less than the lower limit determination value is less than the stop determination number. In other words, the vehicle operation detection device is less likely to stop the gesture determination due to the user's state, the contents of the gesture, and the like. Therefore, the vehicle operation detection device can suppress a decrease in the gesture determination accuracy.

[0013] In the above vehicle operation detection device, it is preferable that the gesture determination unit stops determining the gesture when a magnitude of the displacement vector is equal to or greater than an upper limit determination value. When a user performs a gesture, there is a limit to the speed of movement of a body part. In other words, if the amount of movement of a body part per unit time is excessively large, there is a high possibility that the target of the calculation of the displacement vector is not the user's body part. In this regard, the vehicle operation detection device configured as described above stops judging the gesture when the magnitude of the displacement vector of the body part is equal to or greater than the upper limit judgment value. Therefore, the vehicle operation detection device can reduce the risk of erroneously judging the gesture.

[0014] A vehicle operation detection method for solving the above problem is applied to a vehicle including a vehicle body having an opening, an opening / closing body that opens and closes the opening, a drive unit that operates the opening / closing body to open and close, and a camera that captures images of the periphery of the opening, and detects a user's gesture as a trigger for starting the opening / closing operation of the opening / closing body, the vehicle operation detection method including the steps of: acquiring the position of the body part in the image by inputting the newly captured image into a trained model that has been machine-learned using teacher data that associates the image captured in advance with a body part used in the user's gesture; determining whether the body part has entered a recognition area set within the shooting area of ​​the camera based on the acquired position of the body part; calculating a displacement vector of the body part based on a plurality of the images captured at intervals after it is determined that the body part has entered the recognition area; and determining whether the direction of the displacement vector is in the direction of the recognition area. Did the gesture pattern change in line with the settings? and determining whether the gesture has been performed depending on whether the gesture has been performed or not.

[0015] The vehicle operation detection method having the above-mentioned configuration can obtain the same effects as those of the above-mentioned vehicle operation detection device. Effect of the Invention

[0016] The above-described vehicle operation detection device can improve the detection accuracy of a user's gesture. [Brief description of the drawings]

[0017] [Figure 1] 1 is a schematic diagram of a vehicle including an operation detection device according to an embodiment; [Diagram 2] FIG. 13 is a diagram showing an image captured by a camera, which is an example of teacher data. [Diagram 3] 1 is a schematic diagram showing a state in which a user who has entered a recognition area performs a gesture; [Figure 4] 1 is a schematic diagram showing a state in which a user who has entered a recognition area performs a gesture; [Diagram 5] 4A to 4C are diagrams showing gesture patterns corresponding to the orientation of the user in FIG. 3. [Figure 6] 5A and 5B are diagrams showing gesture patterns corresponding to the user's orientation in FIG. 4. [Figure 7] 4 is a flowchart showing a flow of a process performed by the operation detection device to detect a gesture of a user. [Figure 8] 6 is a flowchart showing a flow of a process performed by the operation detection device to determine whether a user has entered a recognition area. [Figure 9] 6 is a flowchart showing the flow of a process performed by the operation detection device to determine whether or not a user has performed a gesture. [Figure 10] 10 is a flowchart showing a flow of a process performed by an operation detection device according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of a vehicle operation detection device (hereinafter also referred to as an "operation detection device") and a vehicle operation detection method (hereinafter also referred to as an "operation detection method") will be described with reference to the drawings. In the following description, the width direction of the vehicle will be referred to as the "width direction", the front-rear direction of the vehicle will be referred to as the "front-rear direction", and the up-down direction of the vehicle will be referred to as the "up-down direction". In the drawings, an axis extending in the width direction is indicated by the X-axis, an axis extending in the front-rear direction is indicated by the Y-axis, and an axis extending in the up-down direction is indicated by the Z-axis.

[0019] 1, the vehicle 10 includes a vehicle body 20, a front door 30, a sliding door 40, a side mirror 50, and a lock mechanism 60. The vehicle 10 also includes a door actuator 71, a door lock actuator 72, a camera 80, a wireless communication device 90, a door control device 100, and an operation detection device 200.

[0020] The vehicle body 20 has a door opening 21 that is opened and closed by a front door 30, and a door opening 22 that is opened and closed by a sliding door 40. The door openings 21, 22 are areas through which a user passes when moving between the inside and outside of the vehicle 10. The door opening 22 corresponds to an example of an "opening".

[0021] The front door 30 pivots about an axis extending in the vertical direction relative to the vehicle body 20, and is displaced between a fully closed position where the door opening 21 is fully closed and a fully open position where the door opening 21 is fully opened. A side mirror 50 is attached to a portion of the front door 30 near the front end thereof.

[0022] The sliding door 40 slides in the front-rear direction relative to the vehicle body 20, and is displaced between a fully closed position where the door opening 22 is fully closed and a fully open position where the door opening 22 is fully opened. The opening direction of the sliding door 40 is the rearward direction, and the closing direction of the sliding door 40 is the forward direction. The sliding door 40 is opened and closed between the fully closed position and the fully open position by a door actuator 71. The sliding door 40 corresponds to an example of an "opening / closing body", and the door actuator 71 corresponds to an example of a "drive unit".

[0023] The lock mechanism 60 switches between a fully latched state in which the sliding door 40, which is placed in the fully closed position, is restrained to the vehicle body 20, and an unlatched state in which the restraint of the sliding door 40 from the vehicle body 20 in the fully closed position is released. The lock mechanism 60 is shifted from the fully latched state to the unlatched state and from the unlatched state to the fully latched state by a door lock actuator 72. In the following description, the shift of the lock mechanism 60 from the fully latched state to the unlatched state is also referred to as an "unlatch operation", and the shift of the lock mechanism 60 from the unlatched state to the fully latched state is also referred to as a "full latch operation". The door lock actuator 72 corresponds to an example of a "drive unit".

[0024] Camera 80 is installed on side mirror 50 so as to face downward and rearward. Although FIG. 1 illustrates only camera 80 installed on left side mirror 50, camera 80 is also installed on right side mirror 50. As shown in FIG. 1, shooting area A2 of camera 80 includes the area around door opening 22. It is preferable that camera 80 has a wide angle of view. Camera 80 outputs the captured images to operation detection device 200 for each frame.

[0025] The portable device 300 has a switch that is operated when opening / closing or stopping the sliding door 40. The portable device 300 may be a so-called electronic key, a smartphone, or another communication terminal. The wireless communication device 90 wirelessly communicates with the portable device 300 located in the vicinity of the vehicle 10 to determine whether the portable device 300 is associated with the vehicle 10. In this regard, the wireless communication device 90 can determine whether a user carrying the portable device 300 is present within a communication area A1 set in the vicinity of the vehicle 10. The communication area A1 is an area one size larger than the photographing area A2.

[0026] When a switch for operating the sliding door 40 is operated on the portable device 300, the wireless communication device 90 outputs an open operation command signal, a close operation command signal, or a stop command signal to the door control device 100 according to the operated switch. The open operation command signal is a command signal for opening the sliding door 40, and the close operation command signal is a command signal for closing the sliding door 40. The stop command signal is a command signal for stopping the sliding door 40 during opening or closing. Furthermore, when the portable device 300 is present within the communication area A1, the wireless communication device 90 outputs a signal indicating that fact to the operation detection device 200.

[0027] The door control device 100 controls the door actuator 71 and the door lock actuator 72 based on the contents of the input command signal. In detail, when an open operation command signal is input, the door control device 100 performs an unlatch operation on the lock mechanism 60 and then performs an open operation on the sliding door 40. When a close operation command signal is input, the door control device 100 performs a close operation on the sliding door 40 to near the fully closed position and then performs a full latch operation on the lock mechanism 60. When a stop command signal is input, the door control device 100 stops the sliding door 40 that is currently operating.

[0028] Next, the operation detection device 200 will be described. When the operation detection device 200 detects a gesture using the toes T of the user's foot F (hereinafter also referred to as "user's toes T"), it outputs a command signal corresponding to the gesture to the door control device 100. In other words, the user's gesture is a trigger for opening and closing the sliding door 40, and the operation detection device 200 detects the user's request to operate the sliding door 40. In this embodiment, the user's toes T correspond to an example of a "body part" used in the user's gesture.

[0029] As shown in FIG. 1, the operation detection device 200 includes a storage unit 210, an entry determination unit 220, and a gesture determination unit 230. The storage unit 210 stores a trained model that has been machine-learned using training data that associates images captured in advance with the user's toes T. That is, the trained model is a model that receives an image captured by the camera 80 as an input and outputs the position of the user's toes in the image. The trained model is created, for example, when the vehicle 10 is designed, and written to the storage unit 210 when the vehicle 10 is manufactured.

[0030] A method for generating a trained model will be described below. The method for generating a trained model includes a preparation step of preparing training data as shown in Fig. 2, and a training step of performing machine learning based on the training data.

[0031] The preparation process includes an acquisition process for acquiring images of a user standing in the shooting area A2 under various conditions, and a designation process for designating the position of the user's toes in the multiple images acquired in the acquisition process.

[0032] The acquisition process is performed, for example, using a test vehicle associated with the actual vehicle 10. The acquisition process preferably acquires many images captured by changing conditions related to the user and conditions related to the environment around the vehicle 10. The acquisition process preferably acquires images when the user's orientation with respect to the vehicle 10 is different, images when the user's physique is different, images when the user's footwear and clothing are different, images when the user's belongings are different, and images when the direction in which the user's shadow is cast is different. In addition, the acquisition process preferably acquires images when the brightness around the vehicle 10 is different, such as during the day and at night, images when the weather is different, such as during sunny weather and rainy weather, and images when the type of ground on which the vehicle 10 stops is different, such as the presence or absence of pavement. This makes it possible to obtain a trained model that can be adapted to various situations, in other words, a trained model with high versatility.

[0033] In the designation step, the position of the user's toe T is designated in the acquired image, as shown by the dashed arrow in Fig. 2. The position can be designated, for example, using coordinates using pixels in the image. As a result, training data such as that shown in Fig. 2 is generated.

[0034] In the learning process, a model is generated by machine learning using multiple training data as learning data. There are various machine learning methods that can be selected, for example, Convolutional Neural Network (CNN). The trained model outputs the user's toe position by inputting a captured image.

[0035] Next, a description will be given of the entry determination unit 220. The entry determination unit 220 performs an entry determination process and a direction specification process. <Entry judgment processing> In the entry determination process, the entry determination unit 220 determines whether or not the user has entered the recognition area A3 set in the shooting area A2 of the camera 80. In detail, the entry determination unit 220 acquires the user's toe position in the image by inputting images captured at intervals to the trained model. Next, the entry determination unit 220 determines whether or not the acquired toe position is present in the recognition area A3. Since the trained model outputs the toe positions of both feet, the entry determination unit 220 determines whether or not the toe positions of both feet are present in the recognition area A3.

[0036] Then, when the outputted user's toe position is present within the recognition area A3, the entry determination unit 220 determines that the user's toe T has entered the recognition area A3. On the other hand, when the user's toe position is not outputted and when the outputted user's toe position is present outside the recognition area A3, the entry determination unit 220 determines that the user's toe T has not entered the recognition area A3. In the following description, when the entry determination of the user's toe T into the recognition area A3 is positively determined, the image inputted to the trained model is referred to as the "first reference image." In other words, the first reference image can be said to be an image of the user's toe T when the entry determination process is completed.

[0037] 3 and 4 show a state in which the user's toes T, indicated by dashed lines, have entered the recognition area A3. The entry determination unit 220 performs the entry determination process, thereby detecting that the user's toes T have entered the recognition area A3, regardless of the direction of the user's approach to the vehicle 10.

[0038] The orientation of the camera 80 changes depending on whether the side mirror 50 is unfolded or folded. For this reason, it is preferable that the operation detection device 200 adjusts the position of the recognition area A3 depending on whether the side mirror 50 is unfolded or folded. Alternatively, the operation detection device 200 may be configured to adjust the orientation and position of the camera 80 so that the camera 80 faces the same direction regardless of whether the side mirror 50 is unfolded or folded.

[0039] Also, as will be common to the following explanation, depending on the performance of the trained model, the position output by the trained model is not necessarily the position of the user's toes. In other words, the trained model may output the position of a moving object other than the user's toes T. However, in this specification, for ease of understanding of the explanation, it is assumed that the trained model can output the user's toes position when the user's toes position is within the shooting area A2.

[0040] <Direction identification processing> In the direction specification process, the entry determination unit 220 specifies the user's direction Du in the recognition area A3. In detail, the entry determination unit 220 defines the direction from the user's toe position in the first reference image to the user's toe position in the image taken after the first reference image as the user's direction Du. The entry determination unit 220 specifies the user's toe position in the image taken after the first reference image by matching the feature amount of the area including the user's toe position in the first reference image with the image taken after the first reference image. Since the first reference image has two toe positions, the right toe TR and the left toe TL, matching of the feature amounts of both the area including the right toe position and the area including the left toe position is performed. For such matching of the feature amounts between the two images, OpenCV, an open source computer vision library, can be used.

[0041] 3 and 4, the foot F of the user who has advanced through the recognition area A3 is indicated by a solid line, and the user's direction Du is indicated by a solid straight arrow. In other words, the movement direction of the toe T of the user who has advanced through the recognition area A3 is indicated by a solid straight arrow. The entry determination unit 220 can identify the user's direction Du regardless of the user's approach direction to the vehicle 10 by performing a direction identification process. In detail, as shown in FIG. 3, the user's direction Du when the user approaches the vehicle 10 is a direction along the width direction. On the other hand, as shown in FIG. 4, the user's direction Du when the user approaches the vehicle 10 is a direction that intersects with both the width direction and the front-rear direction. When the movement direction of the right toe TR and the movement direction of the left toe TL are in different directions, the sum of the movement directions of both may be regarded as the user's direction Du.

[0042] The entry determination unit 220 may calculate the user's orientation Du based on the first reference image and one image captured after the first reference image, or may calculate the user's orientation Du based on the first reference image and a plurality of images captured after the first reference image. In the following description, the last captured image among the images used to calculate the user's orientation Du is referred to as the "second reference image." In other words, the second reference image is an image of the user's toe T when the direction identification process is completed.

[0043] Next, the gesture determination unit 230 will be described. The gesture determination unit 230 performs a gesture setting process, a displacement vector calculation process, a gesture determination process, and a stop determination process. These processes are performed after it is determined that the user's toe T has entered the recognition area A3.

[0044] <Gesture setting process> In the gesture setting process, the gesture determination unit 230 sets a gesture pattern associated with the user's direction Du. The relationship between the user's direction Du and the gesture pattern will be described below.

[0045] 3 and 4, the gesture for opening the sliding door 40 is to swing the right toe TR so that the right toe TR draws an arc. In other words, the gesture is to open and close the right toe TR without changing the position of the right heel relative to the left foot.

[0046] When the user's direction Du is the direction shown in Fig. 3, the gesture determination unit 230 determines that the user has performed a gesture when the moving direction of the user's right toe TR switches between directions D1 to D8 in the order shown in Fig. 5. On the other hand, when the user's direction Du is the direction shown in Fig. 4, the gesture determination unit 230 determines that the user has performed a gesture when the moving direction of the user's right toe TR switches between directions D1 to D8 in the order shown in Fig. 6.

[0047] As shown in Fig. 3 and Fig. 4, when the user's orientation Du is different, the directions D1 to D8 shown in Fig. 5 and the directions D1 to D8 shown in Fig. 6 are different directions in relation to the vehicle 10, in other words, in relation to the X-axis and Y-axis. As an example, the direction D1 in Fig. 5 is a direction extending in the front-rear direction, whereas the direction D1 in Fig. 6 is a direction intersecting both the width direction and the front-rear direction. In contrast, when the user's orientation Du is used as a reference, the directions D1 to D8 shown in Fig. 5 and the directions D1 to D8 shown in Fig. 6 are the same direction. For example, the direction D1 in Fig. 5 and the direction D1 in Fig. 6 are both inclined 90 degrees clockwise with respect to the user's orientation Du.

[0048] In this way, the gesture determination unit 230 sets a gesture pattern as shown in Figures 5 and 6 in association with the user's direction Du as shown in Figures 3 and 4. Note that the gesture patterns shown in Figures 5 and 6 are merely examples. Therefore, in reality, the gesture patterns may be more detailed or simpler than those shown in Figures 5 and 6.

[0049] <Displacement vector calculation process> In the displacement vector calculation process, the gesture determination unit 230 calculates a displacement vector vd of the user's toe T based on a plurality of images captured at intervals. The direction of the displacement vector vd indicates the movement direction of the user's toe T, and the magnitude of the displacement vector vd indicates the amount of movement of the user's toe T. The displacement vector calculation process is performed after the entry determination unit 220 specifies the user's direction Du.

[0050] The gesture determination unit 230 calculates a displacement vector vd from the user's toe position in the second reference image to the user's toe position in an image captured next to the second reference image (hereinafter also referred to as the "Nth image"). The gesture determination unit 230 identifies the user's toe position in the Nth image by matching the feature amount of an area including the user's toe position in the second reference image with the Nth image. Then, the gesture determination unit 230 calculates the displacement vector vd based on the user's toe positions in both images. Here, the second reference image corresponds to a "specific image" in which the toe position has already been identified, and the Nth image corresponds to an "unidentified image" in which the toe position has not yet been identified.

[0051] Since the second reference image has two toe positions, the right toe TR and the left toe TL, matching of the feature amount is performed for both the area including the right toe position and the area including the left toe position. That is, the gesture determination unit 230 calculates the displacement vector vd of the user's right toe TR and the displacement vector vd of the user's left toe TL. Here, when the user performs a gesture, the user's right toe TR moves, while the user's left toe TL does not move. Therefore, of the two displacement vectors vd calculated by the gesture determination unit 230, the larger displacement vector vd corresponds to the user's right toe TR, and the smaller displacement vector vd corresponds to the user's left toe TL. Therefore, the gesture determination unit 230 does not need to calculate the displacement vector vd of the toe T assumed to be the left toe TL thereafter.

[0052] When the calculation of the displacement vector vd of the user's toe position between the second reference image and the Nth image is completed, the gesture determination unit 230 calculates a displacement vector vd from the user's toe position in the Nth image to the user's toe position in the image captured next to the Nth image (hereinafter also referred to as the "Nth+1th image"). The gesture determination unit 230 identifies the user's toe position in the Nth+1th image by matching the feature amount of the area including the user's toe position in the Nth image with the Nth+1th image. In this case, the Nth image corresponds to the "specific image" and the Nth+1th image corresponds to the "unspecified image". In this way, the gesture determination unit 230 repeatedly calculates the displacement vector vd of the user's toe T between two images captured at a predetermined period every time the next image is acquired from the camera 80. In other words, the "specific image" and the "unspecified image" are switched in sequence.

[0053] The gesture determination unit 230 may calculate the displacement vector vd using an image for each frame output from the camera 80, or may calculate the displacement vector vd using an image for every two frames output from the camera 80. In other words, the gesture determination unit 230 does not need to calculate the displacement vector vd using all images captured by the camera 80. In addition, since the displacement vector vd is calculated for a certain period of time, the magnitude of the displacement vector vd represents the amount of movement of the user's toe T per unit time, in other words, the displacement speed of the user's toe T.

[0054] <Gesture determination process> In the gesture determination process, the gesture determination unit 230 determines that a gesture has been performed when the direction of the displacement vector vd of the user's toe T changes according to a set gesture pattern. That is, the gesture determination unit 230 outputs various command signals to the door control device 100.

[0055] The gesture determination unit 230 may perform the displacement vector calculation process and the gesture determination process simultaneously or sequentially. In the former case, the gesture determination unit 230 determines whether the direction of one displacement vector vd corresponds to a gesture pattern each time the displacement vector vd is calculated. In the latter case, the gesture determination unit 230 determines whether the directions of the multiple displacement vectors vd change to the direction corresponding to the gesture pattern after a set number of displacement vectors vd are calculated.

[0056] <Cancellation decision process> In the stop determination process, the gesture determination unit 230 continues or stops the gesture determination. The stop determination process is a process that is performed in parallel with the gesture determination process.

[0057] For example, when the user does not perform a gesture, such as when the user is stopped in the recognition area A3, the amount of movement of the user's toe T per unit time tends to be "0" or a value close to "0." Therefore, when the magnitude of the displacement vector vd of the toe T is less than a predetermined lower limit judgment value vdth1, the gesture judgment unit 230 stops judging the gesture.

[0058] However, depending on the state when the user performs the gesture, the displacement speed of the user's toe T may temporarily decrease, and the movement amount of the toe T per unit time may temporarily become "0" or a value close to "0". Also, when the direction of the displacement vector vd of the toe T is reversed, the movement amount of the toe T per unit time may temporarily become "0" or a value close to "0". Therefore, the gesture determination unit 230 does not stop the gesture determination when the number of times that the magnitude of the displacement vector vd of the toe T is less than the lower limit judgment value vdth1 is less than a predetermined stop judgment number cntth. In other words, the gesture determination unit 230 stops the gesture determination when the number of times that the magnitude of the displacement vector vd of the toe T is less than the lower limit judgment value vdth1 is equal to or greater than the stop judgment number cntth.

[0059] When a user performs a gesture, there is a limit to the displacement speed of the toe T. In other words, when the amount of movement of the toe T per unit time is excessively large, it is highly likely that the target of the calculation of the displacement vector vd is not the toe T of the user. Therefore, when the magnitude of the displacement vector vd of the toe T is equal to or greater than the upper limit judgment value vdth2, the gesture judgment unit 230 stops judging the gesture.

[0060] It is preferable that the lower limit judgment value vdth1, the upper limit judgment value vdth2, and the stop judgment count cntth are set in advance based on simulations, experiments, etc. For example, it is possible to calculate the average μ and standard deviation σ of the displacement speed based on the measurement results of the displacement speed of the toe T when multiple users perform gestures under multiple conditions. In this case, the lower limit judgment value vdth1 may be set to a value equivalent to "μ-3σ" and the upper limit judgment value vdth2 may be set to a value equivalent to "μ+3σ".

[0061] 7 to 9, a process flow performed by the operation detection device 200 to detect a user's gesture, in other words, an operation detection method, will be described below. The following process is performed in a predetermined control cycle while the vehicle 10 is parked.

[0062] 7, the operation detection device 200 determines whether or not the portable device 300 is present in the communication area A1 based on a signal output from the wireless communication device 90 (S11). If the portable device 300 is not present in the communication area A1 (S11: NO), in other words, if the user is not present in the communication area A1, the operation detection device 200 ends this process.

[0063] On the other hand, when the portable device 300 is present in the communication area A1 (S11: YES), in other words, when the user is present in the communication area A1, the operation detection device 200 activates the camera 80 (S12). In step S12, if it is possible to determine whether the portable device 300 is present in the communication area A1 on the right or left side of the vehicle 10, the camera 80 corresponding to the communication area A1 in which the portable device 300 is present is activated. For example, when the portable device 300 is present in the communication area A1 on the right side of the vehicle 10, the camera 80 provided on the right side mirror 50 is activated.

[0064] Next, the operation detection device 200 performs a process for determining whether the user has entered the recognition area A3 and a process for detecting a gesture by the user (S13, S14). After that, the operation detection device 200 stops the camera 80 (S15) and ends this process.

[0065] Next, the process of determining whether a user has entered the recognition area A3 will be described with reference to the flowchart shown in FIG. As shown in FIG. 8, the operation detection device 200 acquires an image captured by the camera 80 (S21). Next, the operation detection device 200 acquires the user's toe position in the image by inputting the acquired image to the trained model (S22, S23). Next, the operation detection device 200 determines whether the user's toe position is present within the recognition area A3 (S24). If the user's toe position is not present within the recognition area A3 (S24: NO), the operation detection device 200 transitions to the process of step S21. In this case, the operation detection device 200 performs the processes of steps S22 and S23 on the next image. In this way, the operation detection device 200 waits for the user's toe T to enter the recognition area A3.

[0066] On the other hand, when the user's toe position is present within the recognition area A3 (S24: YES), the operation detection device 200 acquires the next image (S25). Here, when the process of step S24 is judged to be positive, the image that was the subject of the judgment corresponds to the first reference image.

[0067] Next, the operation detection device 200 performs matching of features between the first reference image and the image acquired in step S25 (S26). Next, the operation detection device 200 determines whether an area matching the features of the area including the user's toe position in the first reference image is included in the image acquired in step S25 (S27).

[0068] If the matching area is not included in the image acquired in step S25 (S27: NO), the operation detection device 200 proceeds to step S25. On the other hand, if the matching area is included in the image acquired in step S25 (S27: YES), the operation detection device 200 calculates the moving direction of the user's toe T between both images (S28). In step S27, among the images used to calculate the moving direction of the user's toe T, the image that is not the first reference image corresponds to the second reference image. After that, the operation detection device 200 determines the calculated moving direction of the user's toe T as the user's orientation Du within the recognition area A3 (S29).

[0069] Next, with reference to the flowchart shown in FIG. 9, a process for determining whether or not the user has performed a gesture will be described. As shown in Fig. 9, the operation detection device 200 sets a gesture pattern according to the user's direction Du (S31). For example, when the user's direction Du is the direction shown in Fig. 3, the gesture pattern shown in Fig. 5 is set, and when the user's direction Du is the direction shown in Fig. 4, the gesture pattern shown in Fig. 6 is set.

[0070] Next, the operation detection device 200 acquires the next image (S32). Then, the operation detection device 200 performs feature matching between the previously acquired image and the image currently acquired in step S32 (S33). Note that when step S33 is performed for the first time, the previously acquired image becomes the second reference image.

[0071] Next, the operation detection device 200 determines whether or not an area matching the feature amount of the area including the user's toe position in the previously acquired image is included in the currently acquired image (S34). If the matching area is not included in the recognition area A3 in the currently acquired image (S34), the operation detection device 200 proceeds to step S15.

[0072] On the other hand, if the matching area is included in the recognition area A3 in the currently acquired image (S34: YES), the operation detection device 200 calculates a displacement vector vd of the user's toe T (S35). Next, the operation detection device 200 determines whether the magnitude of the displacement vector vd is equal to or greater than the upper limit judgment value vdth2 (S36). If the magnitude of the displacement vector vd is equal to or greater than the upper limit judgment value vdth2 (S36: YES), in other words, if there is a high possibility that the moving object displacing between the two images is not suitable as a gesture detection target, the operation detection device 200 proceeds to step S15.

[0073] On the other hand, if the magnitude of the displacement vector vd is less than the upper limit judgment value vdth2 (S36: NO), the operation detection device 200 judges whether the magnitude of the displacement vector vd is equal to or greater than the lower limit judgment value vdth1 (S37). If the magnitude of the displacement vector vd is less than the lower limit judgment value vdth1 (S37: YES), in other words, if there is a possibility that a moving object displacing between two images is not suitable as a gesture detection target, the operation detection device 200 increments the stop counter cnt. The stop counter cnt is a variable for counting the number of times the magnitude of the displacement vector vd is less than the lower limit judgment value vdth1. The stop counter cnt is initialized to "0" when this process starts.

[0074] Next, the operation detection device 200 judges whether the stop counter cnt is equal to or greater than the stop determination number cntth (S39). If the stop counter cnt is equal to or greater than the stop determination number cntth (S39: YES), for example, if the user remains in the recognition area A3 for a long period of time without performing a gesture, the operation detection device 200 proceeds to step S15. On the other hand, if the stop counter cnt is less than the stop determination number cntth (S39: NO), for example, if the user stops performing a gesture for a very short period of time, the operation detection device 200 proceeds to step S32. In this case, the operation detection device 200 judges whether the user continues the gesture.

[0075] In step S37, if the magnitude of the displacement vector vd is equal to or greater than the lower limit judgment value vdth1 (S37: NO), the operation detection device 200 judges whether or not the direction of the displacement vector vd matches the gesture pattern (S40). If the direction of the displacement vector vd does not match the gesture pattern (S40: NO), the operation detection device 200 shifts the process to step S15. On the other hand, if the direction of the displacement vector vd matches the gesture pattern (S40: YES), the operation detection device 200 judges whether or not the gesture matching is completed (S41). If the gesture matching is not completed (S41: NO), the operation detection device 200 shifts the process to step S32. On the other hand, if the gesture matching is completed (S41: YES), the operation detection device 200 outputs an open operation command signal to the door control device 100 (S42).

[0076] It is preferable that the operation detection device 200 terminates the series of processes in the following cases even after the portable device 300 has entered the communication area A1 (S11: YES): For example, the operation detection device 200 terminates the present process in the following cases: when the portable device 300 has moved out of the communication area A1, when the sliding door 40 has started to open, when the ACC power supply and the IG power supply have been turned on, when a predetermined time-out condition has been established, etc.

[0077] The operation of this embodiment will be described. For example, assume that a user carrying luggage in both hands is about to get into the vehicle 10. In this case, the user moves to the front of the sliding door 40 of the vehicle 10, and then pivots the left foot and swings the toe T of the right foot. Then, the sliding door 40 opens, and the user can place the luggage held in both hands on the back seat. Here, the approach direction of the user to the vehicle 10 may be the front-rear direction of the vehicle 10, the width direction of the vehicle 10, or a direction intersecting both the front-rear direction and the width direction of the vehicle 10. In other words, the user can open the sliding door 40 by performing a certain gesture regardless of the approach direction to the vehicle 10.

[0078] The effects of this embodiment will be described. (1) When the user's toe T enters the recognition area A3, the operation detection device 200 determines whether or not a gesture using the user's toe T has been performed. Here, the operation detection device 200 determines whether or not the user's toe T has entered the recognition area A3 using an image captured by the camera 80 and a trained model. Therefore, it is possible to accurately determine that the user's toe T has entered the recognition area A3. In this way, the operation detection device 200 can improve the detection accuracy of the user's gesture.

[0079] (2) In the displacement vector calculation process, the operation detection device 200 calculates the displacement vector vd of the toe position between multiple images by matching the feature amounts of the area including the toe position. Therefore, the operation detection device 200 can reduce the calculation load when calculating the displacement vector vd.

[0080] (3) When the magnitude of the displacement vector vd of the user's toe T is less than the lower limit determination value vdth1, the operation detection device 200 stops determining the gesture. This can reduce the possibility that the operation detection device 200 will erroneously determine the user's gesture.

[0081] (4) When the number of times that the magnitude of the displacement vector vd of the user's toe T falls below the lower limit judgment value vdth1 is less than the stop judgment count cntth, the operation detection device 200 does not stop the gesture judgment. In other words, the operation detection device 200 is less likely to stop the gesture judgment due to the user's state, the contents of the gesture, and the like. Therefore, the operation detection device 200 can suppress a decrease in the gesture judgment accuracy.

[0082] (5) The operation detection device 200 stops judging the gesture when the magnitude of the displacement vector vd of the user's toe T is equal to or greater than the upper limit judgment value vdth2. This allows the operation detection device 200 to reduce the possibility of erroneously judging the gesture.

[0083] (6) The operation detection device 200 determines that the toe T of the larger displacement vector vd of the two displacement vectors vd of the toe positions of both feet of the user is the toe T performing the gesture. Therefore, even if the user performs a gesture with the left toe TL instead of the right toe TR, the operation detection device 200 can determine that the user has performed the gesture.

[0084] This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction. In the displacement vector calculation process, the gesture determination unit 230 may calculate the displacement vector vd based on the user's toe position for each of a plurality of images obtained by inputting a plurality of images captured at intervals into the trained model. Specifically, the gesture determination unit 230 may calculate the displacement vector vd based on the user's toe position obtained by inputting an image acquired earlier into the trained model and the user's toe position obtained by inputting an image acquired later into the trained model. Hereinafter, a brief description will be given with reference to the flowchart shown in FIG. 10. The flowchart shown in FIG. 10 shows a part different from the flowchart shown in FIG. 9.

[0085] As shown in FIG. 10, the operation detection device 200 acquires an image (S32). Next, the operation detection device 200 acquires the user's toe position by inputting the acquired image into a trained model (S51, S52). After that, the operation detection device 200 determines whether the user's toe position is present within the recognition area A3 (S53). If the user's toe position is not present outside the recognition area A3 (S53: NO), the operation detection device 200 proceeds to step S15. On the other hand, if the user's toe position is present within the recognition area A3 (S53: YES), the operation detection device 200 calculates a displacement vector vd based on the toe position in the image acquired earlier and the toe position acquired later (S33).

[0086] In this way, the gesture determination unit 230 can specify the position of the user's toe in a newly acquired image by inputting the image to the trained model. This allows the operation detection device 200 to accurately calculate the displacement vector vd of the user's toe T. Note that the same processing can also be performed when the entry determination unit 220 performs the direction specification processing (S26, S27).

[0087] The operation detection device 200 may use the above embodiment and the above modified example in combination to obtain the position of the user's toe. The trained model may be a model that receives an image captured by the camera 80 as input and outputs the position of the user's right toe TR. In this case, the teacher data is image data that specifies only the position of the user's right toe TR. The trained model may also include a model that outputs the position of the user's right toe TR and a model that outputs the position of the user's left toe TL.

[0088] The user's gesture may be a gesture using a body part other than the foot F. For example, when the body part is the head, the user's gesture may be a motion of shaking the head up and down or from side to side. When the body part is the hand or arm, the user's gesture may be a motion of shaking the hand or arm up and down or from side to side. When the body part is the eyelid, the user's gesture may be a motion of blinking one eye.

[0089] The operation detection device 200 may perform machine learning in accordance with the user's use of the vehicle 10. This allows the trained model to be adapted to the actual usage environment of the vehicle 10 of the user.

[0090] The operation detection device 200 may detect a user's gesture for closing the sliding door 40. In this case, the gesture for opening the sliding door 40 and the gesture for closing the sliding door 40 may be the same or different.

[0091] The door control device 100 may only perform the unlatching operation of the lock mechanism 60 when the opening operation command signal is input from the operation detection device 200. The camera 80 does not have to be installed on the side mirror 50. For example, the camera 80 may be installed at the upper end of the door opening 22, or may be installed on the sliding door 40.

[0092] The "opening and closing body" may be the front door 30, a back door, or a movable panel of a sunroof device. The door control device 100 and the operation detection device 200 may be configured as one or more processors that operate according to a computer program (software). The door control device 100 and the operation detection device 200 may also be configured as one or more dedicated hardware circuits, such as dedicated hardware (application specific integrated circuit: ASIC) that executes at least some of the various processes. Furthermore, the door control device 100 and the operation detection device 200 may also be configured as a circuit including a combination of these. The processor includes a CPU and memory, such as RAM and ROM. The memory stores program code or instructions configured to cause the CPU to execute processes. The memory, i.e., the storage medium, includes any available medium that can be accessed by a general-purpose or dedicated computer. [Explanation of symbols]

[0093] 10. Vehicle 20…Body 22...Door opening (example of an opening) 40...Sliding door (an example of an opening and closing body) 71...Door actuator (an example of a drive unit) 80…Camera 100...Door control device 200... Vehicle operation detection device 210...Storage section 220...Entry judgment section 230...Gesture determination unit 300…Portable device A1…Communication area A2…Photo area A3…Recognition area T(TR,TL)…Toe vd…Displacement vector vdth1: Lower limit judgment value vdth2: Upper limit judgment value cntth: Stop judgement count

Claims

1. A vehicle operation detection device is applied to a vehicle including a vehicle body having an opening, an opening / closing body that opens and closes the opening, a drive unit that operates the opening / closing body to open and close, and a camera that captures an image of the periphery of the opening, and detects a user's gesture as a trigger for starting the opening / closing operation of the opening / closing body, A storage unit that stores a trained model that is machine-learned using training data that associates the image captured in advance with a body part used in the gesture of the user; an entry determination unit that determines whether or not the body part has entered a recognition area set in a photographing area of ​​the camera based on a position of the body part in the image obtained by inputting the newly photographed image into the trained model; a gesture determination unit that calculates a displacement vector of the body part based on a plurality of images captured at intervals after it is determined that the body part has entered the recognition area, and determines whether the gesture has been performed depending on whether a direction of the displacement vector has changed according to a set gesture pattern. An operation detection device for a vehicle.

2. Among the plurality of images taken at different times, an image in which the position of the body part can be identified is defined as a specific image, and an image in which the position of the body part cannot be identified is defined as an unidentified image. The gesture determination unit performs matching of a feature amount of an area including a position of the body part in the specific image with the unspecified image to specify the position of the body part in the unspecified image, and calculates the displacement vector based on the positions of the body part in the specific image and the unspecified image. The vehicle operation detection device according to claim 1 .

3. The gesture determination unit calculates the displacement vector based on the positions of the body parts for each of the plurality of images obtained by inputting the plurality of images captured at time intervals into the trained model. The vehicle operation detection device according to claim 1 .

4. The gesture determination unit stops determining the gesture when the magnitude of the displacement vector is less than a lower limit determination value. The vehicle operation detection device according to any one of claims 1 to 3.

5. The gesture determination unit does not stop determining the gesture even if the magnitude of the displacement vector is less than the lower limit determination value when the number of times the magnitude of the displacement vector is less than the lower limit determination value is less than a stop determination number. The vehicle operation detection device according to claim 4 .

6. The gesture determination unit stops determining the gesture when the magnitude of the displacement vector is equal to or greater than an upper limit determination value. The vehicle operation detection device according to any one of claims 1 to 5.

7. A vehicle operation detection method is applied to a vehicle including a vehicle body having an opening, an opening / closing body that opens and closes the opening, a drive unit that operates the opening / closing body to open and close, and a camera that captures an image of a periphery of the opening, and detects a user's gesture as a trigger for starting the opening / closing operation of the opening / closing body, A step of acquiring the position of the body part in a newly captured image by inputting the newly captured image into a trained model that is machine-learned using training data that associates the previously captured image with the body part used in the gesture of the user; a step of determining whether or not the body part has entered a recognition area set within a photographing area of ​​the camera based on the acquired position of the body part; calculating a displacement vector of the body part based on a plurality of the images taken at intervals after it is determined that the body part has entered the recognition area; and determining whether the gesture has been performed based on whether the direction of the displacement vector has changed according to a set gesture pattern. A method for detecting operation of a vehicle.

Images