Vehicle gesture detection device and vehicle gesture detection method

Abstract

To provide a gesture detection device for a vehicle and a gesture detection method for a vehicle, which can enhance the detection accuracy of a kicking gesture.SOLUTION: A gesture detection device 120 comprises a determination unit 122 that performs gesture determination processing for determining whether or not a user has performed a kicking gesture on the basis of a position change of a user's foot in a plurality of images shot by a camera 80. The determination unit 122 determines that the user has not performed the kicking gesture when the foot approaching a vehicle body 20 suddenly stops, and continues the gesture determination processing when the foot approaching the vehicle body 20 does not suddenly stop.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a vehicle gesture detection device and a vehicle gesture detection method. [Background technology]

[0002] Patent Document 1 describes a vehicle having a door, a door driving device that drives the door, a sensor that detects a kick gesture using the user's foot around the door, and a door control device that controls the door driving device. When the user makes a kick gesture around the door, the door control device operates the door using the door driving device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-14745 Summary of the Invention [Problem to be solved by the invention]

[0004] When a vehicle is used in cold climates, snow and ice may accumulate on the vehicle's wheel wells, side spoilers, and other parts. In this case, the vehicle user may kick the snow and ice off the vehicle with their feet in order to remove the snow and ice. In other words, the vehicle user may perform a motion similar to a kick gesture. In such a situation, it is preferable that the door control device not recognize the motion of kicking the snow and ice off the vehicle as a kick gesture. [Means for solving the problem]

[0005] The following describes aspects for solving the above problems and their effects. [Aspect 1] A vehicle gesture detection device that solves the above-mentioned problems is applied to a vehicle that includes a vehicle body having a door opening, a door that opens and closes the door opening, a door drive unit that opens and closes the door, and a camera that captures images of the area around the door opening, and detects a user's gesture for opening and closing the door, wherein the gesture is a kick gesture in which the user brings their foot close to the vehicle body and then moves their foot away from the vehicle body, and the vehicle gesture detection device includes a determination unit that performs gesture determination processing to determine whether the user has performed the kick gesture based on changes in the position of the foot in multiple images captured by the camera, and the determination unit determines that the user has not performed the kick gesture if the foot approaching the vehicle body suddenly stops, and continues the gesture determination processing if the foot approaching the vehicle body does not suddenly stop.

[0006] When a user performs a kick gesture or kicks off snow and ice attached to the vehicle body, the user's feet approach the vehicle body and then move away from the vehicle body. In other words, when a user performs a kick gesture or kicks off snow and ice attached to the vehicle body, the user's feet approaching the vehicle body stop temporarily. Here, when a user performs a kick gesture, the user's feet tend to decelerate gradually when they stop. On the other hand, when a user kicks off snow and ice attached to the vehicle body, the user's feet tend to decelerate rapidly when they stop due to hitting the snow and ice.

[0007] Therefore, the vehicle gesture detection device determines that the user has not made a kick gesture when the foot approaching the vehicle body suddenly stops. Therefore, the vehicle gesture detection device can suppress determining that the user has made a kick gesture when the user kicks off snow and ice that have adhered to the vehicle body. In this way, the vehicle gesture detection device can improve the detection accuracy of the kick gesture.

[0008] [Aspect 2] In the vehicle gesture detection device described in Aspect 1, it is preferable that the determination unit determines whether the foot has suddenly stopped based on the deceleration of the foot when the foot approaches the vehicle body and stops.

[0009] The vehicular gesture detection device can easily determine whether the user's feet have suddenly stopped based on the deceleration of the user's feet. [Aspect 3] The vehicular gesture detection device according to aspect 1 or aspect 2 preferably includes a correction unit that corrects the amount of change in the position of the foot in accordance with the distance from the camera to the foot.

[0010] When comparing a case where a user performs a kick gesture close to the camera with a case where a user performs a kick gesture far from the camera, the amount of change in the position of the user's feet between multiple images captured by the camera is likely to be greater in the former case than in the latter case. In this regard, the vehicular gesture detection device includes a correction unit that corrects the amount of change in the position of the user's feet depending on the distance from the camera to the user's feet. Therefore, the vehicular gesture detection device can improve the accuracy of the gesture determination process regardless of the position where the user performs the kick gesture.

[0011] [Aspect 4] In the vehicle gesture detection device described in any one of Aspects 1 to 3, when the foot approaching the vehicle body stops, it is preferable that the judgment unit judges that the user is not performing the kick gesture if the foot is approaching the vehicle body, and continues the gesture judgment process if the position of the foot is away from the vehicle body.

[0012] When a user performs a kick gesture, the user performs the kick gesture with a sufficient gap between the user and the vehicle body so as not to kick the vehicle body. In this regard, the vehicular gesture detection device determines that the user is not performing a kick gesture if the user's feet are close to the vehicle body when the user's feet approaching the vehicle body stop. Therefore, the vehicular gesture detection device can improve the accuracy of the gesture determination process.

[0013] [Aspect 5] In the vehicle gesture detection device described in any one of Aspects 1 to 3, it is preferable that the determination unit determines that the user is not performing the kick gesture if the foot is in contact with the vehicle body when the foot approaches the vehicle body and stops.

[0014] When a user performs a kick gesture, the user performs the kick gesture with a sufficient gap between the user and the vehicle body so as not to kick the vehicle body. In this regard, the vehicular gesture detection device determines that the user is not performing a kick gesture if the user's feet are in contact with the vehicle body when the user's feet approaching the vehicle body stop. Therefore, the vehicular gesture detection device can improve the accuracy of the gesture determination process.

[0015] [Aspect 6] A vehicle gesture detection method that solves the above-mentioned problems is applied to a vehicle that includes a vehicle body having a door opening, a door that opens and closes the door opening, a door drive unit that opens and closes the door, and a camera that captures images of the area around the door opening, and detects a user's gesture for opening and closing the door, wherein the gesture is a kick gesture in which the user brings their foot close to the vehicle body and then moves their foot away from the vehicle body, and the vehicle gesture detection method includes a determination step that performs a gesture determination process that determines whether the user has performed the kick gesture based on changes in the position of the foot in multiple images captured by the camera, and the determination step determines that the user has not performed the kick gesture if the foot approaching the vehicle body suddenly stops, and continues the gesture determination process if the foot approaching the vehicle body does not suddenly stop.

[0016] The vehicular gesture detection method can achieve the same effects as those of the vehicular gesture detection device described above. [Effects of the Invention]

[0017] The vehicle gesture detection device and the vehicle gesture detection method can improve the detection accuracy of a kick gesture performed by a user to open or close a door. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram of a vehicle. [Figure 2] FIG. 2 is an example of an image captured by the camera. [Figure 3] FIG. 3 is a timing chart showing the transition of the displacement amount of the foot and the transition of the velocity of the foot. [Figure 4] FIG. 4 is a flowchart showing the flow of the process performed by the gesture detection device to perform the gesture determination process. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, an embodiment of a vehicle gesture detection device and a vehicle gesture detection method (hereinafter referred to as "vehicle gesture detection device and gesture detection method") will be described with reference to the drawings.

[0020] <Vehicle 10> 1, a vehicle 10 includes a vehicle body 20, a front door 30, a sliding door 40, a door drive unit 50, a lock mechanism 60, a door lock drive unit 70, and a camera 80. The vehicle 10 also includes a wireless communication device 100, a door control device 110, and a gesture detection device 120.

[0021] <Body 20> The vehicle body 20 has a front opening 21 that is opened and closed by a front door 30, and a rear opening 22 that is opened and closed by a sliding door 40. The front opening 21 and the rear opening 22 are areas through which a user passes when moving between the inside and outside of the vehicle 10. In this embodiment, the front opening 21 and the rear opening 22 are separated by a B-pillar of the vehicle body 20. In other embodiments in which the vehicle body 20 does not have a B-pillar, the front opening 21 and the rear opening 22 may be integrated into a single opening. The rear opening 22 corresponds to a "door opening."

[0022] <Front Door 30> The front door 30 has a door body 31, a door handle 32 provided near the rear end of the door body 31, and a side mirror 33 provided near the front end of the door body 31. The front door 30 swings around an axis extending in the vertical direction relative to the vehicle body 20, and is displaced between a fully closed position where the front opening 21 is fully closed and a fully open position where the front opening 21 is fully open.

[0023] <Sliding door 40 and door driving unit 50> The sliding door 40 has a door body 41 and a door knob 42 provided at a position near the front end of the door body 41. The sliding door 40 slides in the front-to-rear direction relative to the vehicle body 20, and is displaced between a fully closed position where the rear opening 22 is fully closed and a fully open position where the rear opening 22 is fully opened. The opening direction of the sliding door 40 is rearward, and the closing direction of the sliding door 40 is forward. The sliding door 40 is opened and closed between the fully closed position and the fully open position by a door drive unit 50. The sliding door 40 can also be called a rear door, since it is located rearward of the front door 30.

[0024] <Lock mechanism 60 and door lock drive unit 70> The locking mechanism 60 switches between a fully latched state in which the sliding door 40, which is positioned in the fully closed position, is restrained to the vehicle body 20, and an unlatched state in which the sliding door 40 is released from the restraint of the sliding door 40 from the vehicle body 20 in the fully closed position. The locking mechanism 60 is transitioned from the fully latched state to the unlatched state and from the unlatched state to the fully latched state by the door lock driving unit 70. In the following description, the transition of the locking mechanism 60 from the fully latched state to the unlatched state is also referred to as an "unlatch operation," and the transition of the locking mechanism 60 from the unlatched state to the fully latched state is also referred to as a "full latch operation." Although not shown in FIG. 1 , the vehicle 10 preferably includes a locking mechanism that restrains the front door 30 to the vehicle body 20.

[0025] <Camera 80> The camera 80 is installed on the side mirror 33 so as to face downward and rearward. As shown in FIG. 1, the photographing area AP of the camera 80 includes the area around the rear opening 22. The camera 80 outputs the photographed images to the gesture detection device 120 for each frame. The camera 80 may be, for example, a camera used in a surroundings monitoring device for autonomous driving and a camera that photographs the surroundings of the vehicle 10.

[0026] <Wireless communication device 100 and portable device 130> The portable device 130 has a switch that is operated to open, close, or stop the sliding door 40. The portable device 130 may be a so-called electronic key, a smartphone, or another communication terminal. The wireless communication device 100 determines whether or not the portable device 130 located in the vicinity of the vehicle 10 is a portable device 130 associated with the vehicle 10 by wirelessly communicating with the portable device 130. In this regard, the wireless communication device 100 can determine whether or not a user carrying the portable device 130 is present within a communication area AC that is set in the vicinity of the vehicle 10. The communication area AC is larger than the photography area AP.

[0027] When a switch for operating the sliding door 40 is operated on the portable device 130, the wireless communication device 100 outputs any one of an open operation command signal, a close operation command signal, and a stop command signal to the door control device 110 according to the operated switch. The open operation command signal is a command signal for opening the sliding door 40, and the close operation command signal is a command signal for closing the sliding door 40. The stop command signal is a command signal for stopping the sliding door 40 during opening or closing. Furthermore, when the portable device 130 is present within the communication area AC, the wireless communication device 100 outputs a signal indicating this to the gesture detection device 120.

[0028] <Door control device 110> The door control device 110 controls the door drive unit 50 and the door lock drive unit 70 based on the content of the input command signal. More specifically, when an open operation command signal is input, the door control device 110 unlatches the lock mechanism 60 and then opens the sliding door 40. When a close operation command signal is input, the door control device 110 closes the sliding door 40 to near the fully closed position and then fully latches the lock mechanism 60. When a stop command signal is input, the door control device 110 stops the sliding door 40 that is currently operating.

[0029] <Gesture detection device 120> When the gesture detection device 120 detects a gesture using the user's foot Ft, it outputs an opening operation command signal to the door control device 110. As shown in FIG. 2 , the gesture in this embodiment is a kick gesture in which the user's foot Ft approaches the vehicle body 20 and then moves the user's foot Ft away from the vehicle body 20. That is, when the user performs a kick gesture, the user's foot Ft approaches the vehicle body 20 in an arc and then moves away from the vehicle body 20 in an arc. The user's foot Ft moves along approximately the same trajectory when the user's foot Ft approaches the vehicle body 20 and when the user's foot Ft moves away from the vehicle body 20. The gesture detection device 120 includes a storage unit 121, a determination unit 122, and a correction unit 123.

[0030] <Storage section 121> The storage unit 121 stores a trained model that has been machine-learned using training data that associates images captured in advance with the user's foot positions Pf. That is, the trained model is a model that receives an image captured by the camera 80 as input and outputs the user's foot positions Pf in the image. The trained model is created, for example, when the vehicle 10 is designed, and written to the storage unit 121 when the gesture detection device 120 is manufactured. In this embodiment, the foot positions Pf refer to the positions of the user's feet Ft, more specifically, the positions of the user's toes.

[0031] Below, we will explain how to generate a trained model. The method for generating a trained model includes a preparation step of preparing training data and a learning step of performing machine learning based on the training data. The preparation step includes an acquisition step of acquiring images of a user standing in a photography area AP under various conditions, and a designation step of designating the user's foot positions Pf in the multiple images acquired in the acquisition step.

[0032] The acquisition step is performed, for example, using an actual vehicle 10. The acquisition step preferably acquires many images captured by changing conditions related to the user and conditions related to the environment around the vehicle 10. This makes it possible to obtain a trained model that can adapt to various situations, in other words, a highly versatile trained model. The designation step designates the user's foot position Pf for the acquired image. The position can be designated using, for example, coordinates using pixels in the image.

[0033] In the learning process, a model is generated by machine learning using multiple training data as learning data. Various machine learning methods can be selected, for example, a convolutional neural network (CNN). When a captured image is input, the trained model outputs the foot position Pf of a person appearing in the image. On the other hand, when an image that does not show the person's feet Ft is input, the trained model does not output the foot position Pf.

[0034] <Determination unit 122> The determination unit 122 performs a start determination process to determine whether or not the user is present near the vehicle 10, and a gesture determination process to determine whether or not the user has performed a kick gesture. In the start determination process and the gesture determination process, the determination unit 122 inputs the image captured by the camera 80 into a trained model, thereby acquiring the foot position Pf in the image.

[0035] In the start determination process, the determination unit 122 determines whether the user's feet Ft are present within a determination area AG set in the image captured by the camera 80. If the user's foot positions Pf are present within the determination area AG, the determination unit 122 performs the gesture determination process. On the other hand, if the determination unit 122 cannot acquire the user's foot positions Pf or if the user's foot positions Pf are not present within the determination area AG, the determination unit 122 does not perform the gesture determination process. The determination area AG is set within the image capture area AP of the camera 80.

[0036] In the gesture determination process, the determination unit 122 calculates a displacement vector of the user's foot Ft between multiple images based on multiple images captured at intervals. Specifically, the determination unit 122 calculates a displacement vector from the foot position Pf in the Nth image to the foot position Pf in the N+1th image. Here, the Nth image is the Nth image captured by the camera 80, and the N+1th image is the N+1th image captured by the camera 80 and is the image in the frame after the Nth image.

[0037] For example, the determination unit 122 acquires the foot position Pf in the N+1th image corresponding to the foot position Pf in the Nth image by matching the feature amount of the area including the foot position Pf in the Nth image with respect to the N+1th image. The determination unit 122 may acquire the foot position Pf in the N+1th image by inputting the N+1th image to a trained model. Then, the determination unit 122 calculates a displacement vector of the foot position Pf based on the foot position Pf in both images. Next, the determination unit 122 calculates a displacement vector from the foot position Pf in the N+1th image to the foot position Pf in the N+2th image. Furthermore, the determination unit 122 calculates a displacement vector from the foot position Pf in the N+2th image to the foot position Pf in the N+3th image.

[0038] In this way, the determination unit 122 calculates the displacement vector of the foot position Pf each time a new image is captured. The determination unit 122 may calculate the displacement vector of the foot position Pf for each frame, or may calculate the displacement vector of the foot position Pf for each set of frames. The direction of the displacement vector of the foot position Pf indicates the direction of movement of the user's foot Ft. The magnitude of the displacement vector of the foot position Pf indicates the amount of displacement of the user's foot Ft per time period corresponding to a predetermined number of frames, i.e., the speed of the user's foot Ft. Therefore, the determination unit 122 calculates the speed of the user's foot Ft when calculating the displacement vector of the foot position Pf.

[0039] The determination unit 122 determines that a kick gesture has been performed when the direction of the displacement vector of the foot position Pf changes in accordance with a gesture pattern corresponding to the kick gesture. In this embodiment, the determination unit 122 alternately calculates the displacement vector of the foot position Pf and checks the gesture pattern. In another embodiment, the determination unit 122 may simultaneously calculate the displacement vector of the foot position Pf and check the gesture pattern. When the determination unit 122 determines that the user has performed a kick gesture within the determination area AG, it outputs an open operation command signal to the door control device 110.

[0040] Incidentally, when the vehicle 10 is used in cold regions, snow and ice may accumulate around the wheel wells and on the side spoilers of the vehicle body 20. In this case, the user may perform an action of kicking off the snow and ice with their feet (hereinafter also referred to as a "removal action") in order to remove the snow and ice adhering to the vehicle body 20. The movement of the user's foot Ft when performing the removal action is equivalent to the movement of the user's foot Ft when performing a kick gesture. More specifically, the change in the direction of the displacement vector of the user's foot position Pf when performing the removal action is equivalent to the change in the direction of the displacement vector of the foot position Pf when performing a kick gesture. Therefore, it is preferable that the gesture detection device 120 does not determine that the user has performed a kick gesture when performing a removal action. Note that the removal action may be an action in which the user kicks a tire in order to remove mud or the like from shoes.

[0041] Next, referring to Fig. 3, changes in the displacement amount and speed of the user's foot Ft when the user performs a kick gesture and when the user performs a removal operation will be described. Here, the axis extending in the longitudinal direction of the vehicle 10 is defined as the X-axis, the axis extending in the width direction of the vehicle 10 is defined as the Y-axis, and the axis extending in the vertical direction of the vehicle 10 is defined as the Z-axis. In this case, the displacement amount and speed of the user's foot Ft refer to the three-dimensional displacement amount and speed of the user's foot Ft in the XYZ space. In addition, the foot position Pf immediately before the kick gesture and removal operation are started is defined as the origin.

[0042] First, a case where the user performs a kick gesture will be described. As shown by the solid line in FIG. 3, when a user performs a kick gesture, the displacement of the user's foot Ft gradually increases from a first timing t11 when the user starts the kick gesture. Also, the velocity of the user's foot Ft gradually increases from the first timing t11. Between the first timing t11 and the following second timing t12, the velocity of the user's foot Ft switches from an increasing trend to a decreasing trend. Subsequently, at the second timing t12, the velocity of the user's foot Ft becomes "0". During the period from the first timing t11 to the second timing t12, the velocity of the user's foot Ft is greater than "0", and therefore, the displacement of the user's foot Ft continues to increase during the same period.

[0043] At the second timing t12, the direction of movement of the user's foot Ft is reversed. Therefore, from the second timing t12, the velocity of the user's foot Ft gradually decreases. Between the second timing t12 and the next third timing t13, the velocity of the user's foot Ft switches from a decreasing trend to an increasing trend. Subsequently, at the third timing t13, the velocity of the user's foot Ft becomes "0". Furthermore, the user's foot Ft returns to the position it was in when the kick gesture began. During the period from the second timing t12 to the third timing t13, the velocity of the user's foot Ft is less than "0", so the displacement amount of the user's foot Ft continues to decrease during that period.

[0044] Next, a case where the user performs a removal operation will be described. As shown by the dashed-dotted line in FIG. 3 , when the user performs the removal operation, the displacement of the user's feet Ft gradually increases from the first timing t21 when the user starts the removal operation. Also, the speed of the user's feet Ft gradually increases from the first timing t21. Thereafter, at the second timing t22, the user's feet Ft come into contact with snow and ice adhering to the vehicle body 20, causing the user's feet Ft to suddenly stop. That is, at the second timing t22, the speed of the user's feet Ft becomes "0" and the displacement of the user's feet Ft stops increasing. Then, at the third timing t23, the user begins to return the kicked-up feet Ft toward their original positions. Therefore, from the third timing t23, the displacement of the user's feet Ft gradually decreases, and the speed of the user's feet Ft also gradually decreases. Between the third timing t23 and the following fourth timing t24, the velocity of the user's foot Ft switches from a decreasing trend to an increasing trend. Then, at the fourth timing t24, the velocity of the user's foot Ft becomes "0." Also, the user's foot Ft returns to the position it was in when the removal operation started.

[0045] 3, when the user performs a kick gesture, the user's foot Ft approaching the vehicle body 20 gradually decelerates and stops at the second timing t12. In other words, at the second timing t12, the deceleration, which is the gradient of the speed of the user's foot Ft, is relatively small. Therefore, at the timing t12* immediately before the second timing t12, the speed of the user's foot Ft is slower than the second determination speed Vth2.

[0046] 3, when the user performs the removal work, the user's feet Ft approaching the vehicle body 20 suddenly decelerate and stop at the second timing t22. In other words, at the second timing t22, the deceleration, which is the gradient of the speed of the user's feet Ft, is relatively large. Therefore, at timing t22* immediately before the second timing t22, the speed of the user's feet Ft is greater than the second determination speed Vth2.

[0047] Therefore, the determination unit 122 continues the gesture determination process if the user's feet Ft approaching the vehicle body 20 do not suddenly stop. On the other hand, the determination unit 122 determines that the user is not performing a kick gesture if the user's feet Ft approaching the vehicle body 20 suddenly stop. In this case, the determination unit 122 stops the gesture determination process.

[0048] The determination unit 122 determines whether the user's feet Ft have suddenly stopped based on the deceleration of the user's feet Ft when the user's feet Ft approaching the vehicle body 20 stop. In this embodiment, the determination unit 122 determines whether the user's feet Ft have suddenly stopped based on the speed of the user's feet Ft at a timing (hereinafter referred to as the "immediately preceding timing") that is a predetermined number of frames prior to the timing at which the user's feet Ft stopped. The predetermined frame may be any frame that corresponds to the calculation interval of the displacement vector of the user's foot position Pf.

[0049] The determination unit 122 determines that the user's foot Ft has suddenly stopped if the velocity of the user's foot Ft at the timing immediately before the timing when the user's foot Ft stopped is equal to or greater than the second determination velocity Vth2. On the other hand, the determination unit 122 determines that the user's foot Ft has not suddenly stopped if the velocity of the user's foot Ft at the timing immediately before the timing when the user's foot Ft stopped is less than the second determination velocity Vth2. Here, the purpose of comparing the velocity of the user's foot Ft at the immediately previous timing with the second determination velocity Vth2 is to determine whether there was a large change in the velocity of the user's foot Ft immediately before the user's foot Ft stopped. In this respect, the comparison can also be considered a determination based on the deceleration of the user's foot Ft. The second determination velocity Vth2 is preferably determined based on measurement results of the velocity of the foot Ft when the user performs a kick gesture and when the user performs a removal motion.

[0050] In this embodiment, the gesture detection device 120 determines that the user's feet Ft have stopped when the speed of the user's feet Ft approaching the vehicle body 20 is less than the first determination speed Vth1. That is, the first determination speed Vth1 is a speed for determining whether the feet Ft have stopped during deceleration, and therefore the first determination speed Vth1 is preferably set to a value less than the second determination speed Vth2 and equal to or slightly greater than "0".

[0051] When the user performs a kick gesture, the user performs the kick gesture so that the feet Ft do not come into contact with the vehicle body 20. Therefore, when the user performs a kick gesture, even when the user's feet Ft are closest to the vehicle body 20, the distance between the user's feet Ft and the vehicle body 20 is likely to be large. On the other hand, when the user performs a removal operation, the user's feet Ft are likely to come close to the vehicle body 20 in order to kick off snow and ice adhering to the vehicle body 20. Therefore, when the user performs a removal operation, the distance between the user's feet Ft and the vehicle body 20 is likely to be small when the user's feet Ft are closest to the vehicle body 20. Furthermore, depending on the locations on the vehicle body 20 where snow and ice adhere, when the user performs a removal operation, the user's feet Ft may come into contact with the vehicle body 20 in the image captured by the camera 80.

[0052] Therefore, the determination unit 122 continues the gesture determination process if the user's foot position Pf is away from the vehicle body 20 when the user's foot position Pf approaching the vehicle body 20 stops. On the other hand, the determination unit 122 determines that the user is not performing a kick gesture if the user's foot Ft is approaching the vehicle body 20 when the user's foot Ft approaching the vehicle body 20 stops. Furthermore, the determination unit 122 determines that the user is not performing a kick gesture if the user's foot Ft is in contact with the vehicle body 20 when the user's foot Ft approaching the vehicle body 20 stops.

[0053] <Correction unit 123> A user standing closer to the camera 80 is likely to appear larger in the image captured by the camera 80 than a user standing farther from the camera 80. In other words, when a user moves their foot Ft at a position close to the camera 80, the displacement vector of the foot position Pf is likely to become large. On the other hand, when a user moves their foot Ft at a position farther from the camera 80, the displacement vector of the foot position Pf is likely to become small. The magnitude of the displacement vector of the foot position Pf corresponds to the "amount of change in the position of the foot."

[0054] Therefore, the correction unit 123 corrects the magnitude of the displacement vector of the foot position Pf between multiple images captured by the camera 80, depending on the distance from the camera 80 to the user's feet Ft. As a result, the speed of the user's feet Ft is corrected depending on the distance from the camera 80 to the user's feet Ft.

[0055] <Gesture detection method> The following describes the flow of processing, i.e., the gesture detection method, performed by the gesture detection device 120. This processing is performed in a predetermined control cycle when the user carrying the portable device 130 enters the communication area AC and the sliding door 40 is in the fully closed position.

[0056] 4, the gesture detection device 120 acquires an image captured by the camera 80 (S11). Subsequently, the gesture detection device 120 acquires a foot position Pf by inputting the image captured by the camera 80 into a trained model stored in the storage unit 121 (S12). After that, the gesture detection device 120 determines whether the foot position Pf in the image is present within the determination area AG (S13). If the foot position Pf is not present within the determination area AG (S13: NO), the gesture detection device 120 ends this process. If the foot position Pf cannot be acquired in step S12, the gesture detection device 120 also ends this process.

[0057] On the other hand, if the foot position Pf is present within the determination area AG (S13: YES), the gesture detection device 120 acquires a new image captured by the camera 80 (S14). The image acquired in step S14 is an image captured after the previously acquired image. Next, the gesture detection device 120 acquires the foot position Pf in the new image (S15).

[0058] Then, the gesture detection device 120 calculates a displacement vector of the foot position Pf between frames of the camera 80 (S16). The displacement vector of the foot position Pf is a vector directed from the previously identified foot position Pf to the currently identified foot position Pf. It is preferable that the displacement vector of the foot position Pf has been corrected according to the distance from the camera 80 to the user's foot Ft.

[0059] The gesture detection device 120 determines whether or not the user's feet Ft are approaching the vehicle body 20 based on the direction of the displacement vector of the foot position Pf (S17). If the user's feet Ft are not approaching the vehicle body 20 (S17: NO), that is, if the displacement vector of the foot position Pf is not pointing toward the vehicle body 20, the gesture detection device 120 proceeds to step S21, which will be described later. On the other hand, if the user's feet Ft are approaching the vehicle body 20 (S17: YES), that is, if the displacement vector of the foot position Pf is pointing toward the vehicle body 20, the gesture detection device 120 determines whether or not the speed of the user's feet Ft is less than the first determination speed Vth1 (S18). As described above, the speed of the user's feet Ft corresponds to the magnitude of the displacement vector of the foot position Pf calculated in step S16.

[0060] If the speed of the user's foot Ft is equal to or greater than the first determination speed Vth1 (S18: NO), in other words, if the user's foot Ft is moving, the gesture detection device 120 proceeds to step S21, which will be described later. On the other hand, if the speed of the user's foot Ft is less than the first determination speed Vth1 (S18: YES), in other words, if the user's foot Ft has stopped, the gesture detection device 120 determines whether the previous speed of the user's foot Ft is equal to or greater than the second determination speed Vth2 (S19). The previous speed of the user's foot Ft is not the speed calculated in the current loop processing, but the speed calculated in the previous loop processing. In other words, the previous speed of the user's foot Ft is the speed at the timing immediately before the user's foot Ft stopped.

[0061] If the speed of the user's feet Ft is equal to or greater than the second determination speed Vth2 (S19: YES), in other words, if the user's feet Ft have suddenly stopped, the gesture detection device 120 ends this process. In this case, it is determined that the user is performing a removal operation. On the other hand, if the speed of the user's feet Ft is less than the second determination speed Vth2 (S19: NO), in other words, if the user's feet Ft have not suddenly stopped, the gesture detection device 120 determines whether or not the foot position Pf when the user's feet Ft have stopped is separated from the vehicle body 20 (S20). At this time, the gesture detection device 120 may calculate the distance from the user's foot position Pf to the vehicle body 20. Then, the gesture detection device 120 may compare the distance from the user's foot position Pf to the vehicle body 20 with a predetermined determination distance to determine whether or not the foot position Pf when the user's feet Ft have stopped is separated from the vehicle body 20.

[0062] If the foot position Pf is not separated from the vehicle body 20 (S20: NO), the gesture detection device 120 ends this process. On the other hand, if the foot position Pf is separated from the vehicle body 20 (S20: YES), the gesture detection device 120 determines whether or not the displacement vector of the foot position Pf matches a gesture pattern (S21). If the displacement vector of the foot position Pf does not match a gesture pattern (S21: NO), the gesture detection device 120 ends this process.

[0063] On the other hand, if the displacement vector of the foot position Pf matches the gesture pattern (S21: YES), the gesture detection device 120 determines whether or not matching of the gesture pattern is complete (S22). If matching of the gesture pattern is not complete (S22: NO), the gesture detection device 120 proceeds to step S14. On the other hand, if matching of the gesture pattern is complete (S22: YES), the gesture detection device 120 outputs an open operation command signal to the door control device 110 (S23). Thereafter, the gesture detection device 120 ends this process.

[0064] In the flowchart described above, steps S14 to S22 correspond to the "determination step." <Actions and Effects of This Embodiment> (1) When the user performs a kick gesture, the user's foot Ft approaching the vehicle body 20 does not suddenly stop, as shown by the solid line in FIG. 3 . If the user's foot Ft does not suddenly stop, the gesture detection device 120 continues the gesture determination process. Therefore, the sliding door 40 is opened after the user has finished performing the kick gesture. On the other hand, when the user performs a removal operation, the user's foot Ft approaching the vehicle body 20 suddenly stops, as shown by the dashed-dotted line in FIG. 3 . If the user's foot Ft suddenly stops, the gesture detection device 120 determines that the user is not performing a kick gesture. Therefore, even if the user has finished performing the removal operation, the sliding door 40 is prevented from being opened. In this way, the gesture detection device 120 can improve the detection accuracy of the kick gesture for opening the sliding door 40.

[0065] (2) The gesture detection device 120 determines whether the user's feet Ft approaching the vehicle body 20 have stopped by comparing the speed of the user's feet Ft with the first determination speed Vth1. The gesture detection device 120 also determines whether the user's feet Ft have suddenly stopped by comparing the speed of the user's feet Ft at the timing immediately before the user's feet Ft stop with the second determination speed Vth2. Therefore, the gesture detection device 120 can easily determine whether the user's feet Ft have stopped and whether the user's feet Ft have suddenly stopped.

[0066] (3) The gesture detection device 120 corrects the displacement vector of the user's foot position Pf, in other words, the velocity of the user's foot Ft, depending on the distance from the camera 80 to the user's foot Ft. Therefore, the gesture detection device 120 can improve the accuracy of the gesture determination process regardless of the position where the user performs the kick gesture.

[0067] (4) When the user's foot Ft approaches the vehicle body 20 and stops, the gesture detection device 120 determines that the user is not performing a kick gesture if the user's foot Ft is approaching the vehicle body 20. Therefore, the gesture detection device 120 can improve the accuracy of the gesture determination process.

[0068] (5) When the user's foot Ft approaches the vehicle body 20 and stops, the gesture detection device 120 determines that the user is not performing a kick gesture if the user's foot Ft is in contact with the vehicle body 20. Therefore, the gesture detection device 120 can improve the accuracy of the gesture determination process.

[0069] <Example of change> This embodiment can be modified as follows: This embodiment and the following modifications can be combined and implemented within the scope of technical compatibility.

[0070] 3, when the user performs a removal action, the speed of the user's foot Ft approaching the vehicle body 20 tends to increase. Therefore, the gesture detection device 120 may determine that the user is not performing a kick gesture when the speed of the user's foot Ft approaching the vehicle body 20 is equal to or greater than a predetermined upper limit determination speed.

[0071] The gesture detection device 120 may acquire the outside air temperature of the vehicle 10. If the outside air temperature of the vehicle 10 is a temperature at which snow and ice will not accumulate on the vehicle body 20, the gesture detection device 120 may not need to perform the processes of steps S17 to S20 in the flowchart shown in FIG.

[0072] The determination area AG may be limited to only the area adjacent to the rear opening 22 in the width direction. In this case, the user will perform the kick gesture within a very limited area. Therefore, the gesture detection device 120 does not need to make corrections according to the distance from the camera 80 to the user's feet Ft.

[0073] The determination area AG may be divided into a plurality of determination areas AG that are divided according to the distance from the camera 80. Here, it is assumed that the determination area AG is divided into a first area that is close to the camera 80 and a second area that is far from the camera 80. When the user's feet Ft are located in the first area, the correction unit 123 does not need to perform correction according to the distance from the camera 80 to the user's feet Ft. On the other hand, when the user's feet Ft are located in the second area, the correction unit 123 may perform correction according to the distance from the camera 80 to the user's feet Ft.

[0074] The gesture detection device 120 may calculate the deceleration of the user's feet Ft when the user's feet Ft approaching the vehicle body 20 stop. Then, the gesture detection device 120 may determine whether the user's feet Ft have suddenly stopped by comparing the deceleration of the user's feet Ft with a predetermined determination deceleration.

[0075] The gesture detection device 120 does not need to determine whether the user is performing a kick gesture based on the positional relationship between the vehicle body 20 and the stopping position of the user's foot Ft approaching the vehicle body 20. That is, step S20 may be omitted from the flowchart shown in FIG.

[0076] The gesture detection device 120 does not need to include the correction unit 123. In the above embodiment, the gesture detection device 120 outputs an opening operation command signal to the door control device 110 for opening the sliding door 40 based on a kick gesture of the user. In contrast, the gesture detection device 120 according to a modified example may output a closing operation command signal to the door control device 110 for closing the sliding door 40 based on a kick gesture of the user. Furthermore, the gesture detection device 120 according to another modified example may output one of an opening operation command signal and a closing operation command signal to the door control device 110 depending on the position of the sliding door 40 when the user performs a kick gesture.

[0077] The vehicle 10 may include a front door drive unit that drives the front door 30. In this case, the gesture detection device 120 may detect a kick gesture for opening the front door 30, similar to the case of opening the sliding door 40.

[0078] The vehicle 10 may include a back door that opens and closes an opening at the rear of the vehicle body 20, a back door drive unit that drives the back door, and a backup camera that captures images behind the vehicle 10. In this case, the gesture detection device 120 may detect a kick gesture for opening the back door, similar to when opening the sliding door 40.

[0079] The door control device 110 and the gesture detection device 120 may be configured as one or more processors operating according to a computer program (software). The door control device 110 and the gesture detection device 120 may also be configured as one or more dedicated hardware circuits, such as dedicated hardware (application-specific integrated circuits: ASICs) that execute at least some of the various processes. Furthermore, the door control device 110 and the gesture detection device 120 may also be configured as circuits that include a combination of these. The processor includes a CPU and memory, such as RAM and ROM. The memory stores program code or instructions configured to cause the CPU to execute processes. The memory, i.e., storage medium, includes any available medium accessible by a general-purpose or dedicated computer. [Explanation of symbols]

[0080] AC: Communication area AG: Judgment area AP: Photo area Ft...foot Pf…foot position 10...Vehicle 20...Body 22...Rear opening (door opening) 40...Sliding door (door) 50...Door drive unit 80...Camera 110...Door control device 120...Gesture detection device 121...Storage section 122…Judgment section 123...Correction section

Claims

1. A vehicle gesture detection device is applied to a vehicle including a vehicle body having a door opening, a door that opens and closes the door opening, a door drive unit that opens and closes the door, and a camera that photographs the periphery of the door opening, and detects a user's gesture for opening and closing the door, the gesture is a kick gesture in which the user moves their foot closer to the vehicle body and then moves their foot away from the vehicle body; a determination unit that performs a gesture determination process to determine whether or not the user has performed the kick gesture based on a change in the position of the foot in a plurality of images captured by the camera; The determination unit determines that the user has not performed the kick gesture when the foot approaching the vehicle body suddenly stops, and continues the gesture determination process when the foot approaching the vehicle body does not suddenly stop. Gesture detection device for vehicles.

2. The determination unit determines whether the foot has suddenly stopped based on the deceleration of the foot when the foot approaching the vehicle body stops. The gesture detection device for a vehicle according to claim 1 .

3. a correction unit that corrects the amount of change in the position of the foot in accordance with the distance from the camera to the foot; The gesture detection device for a vehicle according to claim 1 .

4. When the foot approaching the vehicle body stops, the determination unit determines that the user is not performing the kick gesture if the foot is approaching the vehicle body, and continues the gesture determination process if the foot is away from the vehicle body. The gesture detection device for a vehicle according to any one of claims 1 to 3.

5. The determination unit determines that the user is not performing the kick gesture if the foot is in contact with the vehicle body when the foot approaching the vehicle body stops. The gesture detection device for a vehicle according to any one of claims 1 to 3.

6. A vehicle gesture detection method is applied to a vehicle including a vehicle body having a door opening, a door that opens and closes the door opening, a door drive unit that opens and closes the door, and a camera that photographs the periphery of the door opening, and detects a user's gesture for opening and closing the door, the gesture is a kick gesture in which the user moves their foot closer to the vehicle body and then moves their foot away from the vehicle body; a determination step of performing a gesture determination process to determine whether or not the user has performed the kick gesture based on a change in the position of the foot in a plurality of images captured by the camera; The determination step determines that the user has not performed the kick gesture if the foot approaching the vehicle body suddenly stops, and continues the gesture determination process if the foot approaching the vehicle body does not suddenly stop. A gesture detection method for a vehicle.

Images