Method, apparatus, electronic device, and storage medium for detecting sheesh gestures

Abstract

This application provides a method, apparatus, electronic device, and storage medium for detecting see gestures. This application, implemented by gesture, contributes to improved communication efficiency because it can complete control more quickly than the prior art. Furthermore, the gesture method is less susceptible to environmental influences than voice control methods, contributing to improved interference resistance in vehicle cabin environments.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer technology, and specifically relates to a gesture detection method, apparatus, electronic device and storage medium.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority based on the Chinese application filed with the China National Intellectual Property Administration on August 26, 2024, with application number 202411170859.7 and titled "Gesture Detection Method, Apparatus, Electronic Device and Storage Medium", the entire content of which is incorporated into this application by reference.

BACKGROUND ART

[0003] Currently, interaction for some functions of a vehicle, such as pausing music or opening and closing a sunroof, is performed via control buttons (including physical buttons and touch buttons). When a user controls the above functions, it is necessary to use the corresponding button. In such a control method, control cannot be performed without locating the corresponding button. Although a voice control system exists as a complementary means, when there is noise in the vehicle cabin, the accuracy and response speed of voice recognition will decrease significantly. Therefore, an interaction method with excellent anti-interference performance and easy interaction is urgently required at present.

SUMMARY OF THE INVENTION

[0004] In view of the above, the embodiments of the present application provide a gesture detection method, apparatus, electronic device and storage medium that improve the anti-interference performance of interaction in a vehicle cabin environment and enable efficient interaction.

[0005] In a first aspect, the embodiments of the present application provide a gesture detection method, the method comprising: acquiring facial feature points in a face image based on the face image including the in-vehicle face captured by an in-vehicle camera; The steps include: extracting a first target image from the face image based on a rectangular region constructed with the target feature point among the face feature points as the center, where the width is the value obtained by multiplying the area of ​​the bounding rectangle of the figure composed of the face feature points by a first preset multiplier, the height is the value obtained by multiplying the width by a preset ratio, and the direction of extension of the face in the face image is the direction of slope of the height; The steps include inputting the first target image into a sheesh detection model and determining whether or not the face image contains a sheesh gesture, If it is determined that the facial image includes the sheesh gesture, the step of controlling a target controllable component in the vehicle to perform a pre-setting operation is included, The first pre-setting multiplier is a positive number, the pre-setting ratio is less than 1, and the target feature point belongs to the feature point of the lip region.

[0006] After optionally acquiring the facial feature points, the method proceeds as follows: The steps include inputting the aforementioned face image into a face detection model and obtaining initial face feature points in the face image, The steps include: mapping the initial facial feature points to the same feature points in a standard three-dimensional facial model to obtain a reconstructed three-dimensional facial model; The process further includes the step of mapping the reconstructed three-dimensional face model to the same feature points in the face image and determining the target feature points.

[0007] The step of controlling a target-controllable component in the vehicle to perform a pre-setting operation if it is optionally determined that the facial image includes the sheesh gesture is: If it is determined that the face image contains the "shush" gesture, the steps include: extracting a second target image from the face image based on a rhombus constructed with a point at a predetermined distance from the target feature point along the opposite direction of extension, with the side length being the length of the face multiplied by a second preset multiplier, and the extension direction being the inclination direction; Based on the second target image, the steps include determining a virtual "shush" slot in the region where the index finger is positioned when making a "shush" gesture in the second target image, The steps include inputting the aforementioned second target image into the hand feature point positioning model and obtaining hand feature points, The steps include determining whether the "shush" gesture is a correct "shush" gesture based on the index finger feature point representing the index finger among the hand feature points, and the positional relationship between the index finger feature point and the "shush" virtual slot, The process includes the step of controlling a target controllable component in the vehicle to perform a pre-set operation if the see gesture is correct.

[0008] Optionally, the target feature point is a lip feature point representing the center of the upper lip, and the three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the three-dimensional face reconstruction model as the origin, the direction of the line connecting both eyes as the X-axis, the face orientation as the Z-axis, and the direction perpendicular to the plane formed by the X-axis and Z-axis as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system.

[0009] Optionally, the length of the shee virtual slot is the distance from the nasal tip feature point, which represents the tip of the nose among the facial feature points, to the lip feature point, the upper end point of the shee virtual slot is located at a point moved 0.1 times the length from the nasal tip feature point along the positive Z-axis direction, and the lower end point of the shee virtual slot is located at a point moved 0.1 times the length from the lip feature point along the positive Z-axis direction.

[0010] Optionally, the predetermined distance is 0.25 times the height of the face, and the second preset multiplier is a numerical value of 1.

[0011] Optionally, the first preset multiplier is the square root of the area of ​​the circumscribed rectangle, and the preset ratio is two-thirds.

[0012] The step of optionally controlling a target-controllable component in the vehicle to perform a pre-set operation is: If the target controllable component is an in-vehicle phone, the step of rejecting the current incoming call on the in-vehicle phone, If the target controllable component is an in-vehicle multimedia system, the steps include: pausing the current playback content of the in-vehicle multimedia system; If the target controllable component is a car window, the method includes the step of closing the car window.

[0013] In the second phase, the embodiments of this application provide a see gesture detection device, the device is An acquisition unit that acquires facial feature points in a facial image, including faces inside the vehicle, based on a facial image captured by an in-vehicle camera, An extraction unit extracts a first target image from a face image based on a rectangular region constructed with a target feature point among the face feature points as the center, where the width is the value obtained by multiplying the area of ​​the bounding rectangle of the figure composed of the face feature points by a first preset multiplier, the height is the value obtained by multiplying the width by a preset ratio, and the direction of extension of the face in the face image is the direction of the slope of the height. A determination unit inputs the first target image into a shhh detection model and determines whether or not the face image contains a shhh gesture, A control unit that controls a target-controllable component in the vehicle to perform a pre-setting operation if it is determined that the facial image includes the sheesh gesture, In the extraction unit, the first pre-setting multiplier is a positive number, the pre-setting ratio is less than 1, and the target feature point belongs to the feature point of the lip region.

[0014] Optionally, the apparatus further includes a processing unit which, after acquiring the facial feature points, inputs the facial image into a face detection model, acquires initial facial feature points in the facial image, maps the initial facial feature points to the same feature points in a standard three-dimensional face model to acquire a reconstructed three-dimensional face model, maps the reconstructed three-dimensional face model to the same feature points in the facial image to determine the target feature points.

[0015] Optionally, if the control unit determines that the facial image includes the sheesh gesture, it controls the target controllable component in the vehicle to perform a pre-setting operation. If it is determined that the face image contains the "shush" gesture, a second target image is extracted from the face image based on a rhombus constructed with a point at a predetermined distance from the target feature point along the opposite direction of extension, where the side length is the length of the face multiplied by a second preset multiplier, and the extension direction is the inclination direction. Based on the second target image, determine the virtual "shush" slot in the region where the index finger is positioned when making a "shush" gesture in the second target image, The aforementioned second target image is input into the hand feature point positioning model to obtain hand feature points, Based on the index finger feature point representing the index finger among the hand feature points, and the positional relationship between the index finger feature point and the virtual shhh slot, it is determined whether the shhh gesture is a correct shhh gesture. This includes controlling a target controllable component in the vehicle to perform a pre-set action if the correct see gesture is detected.

[0016] Optionally, the target feature point is a lip feature point representing the center of the upper lip, and the three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the three-dimensional face reconstruction model as the origin, the direction of the line connecting both eyes as the X-axis, the face orientation as the Z-axis, and the direction perpendicular to the plane formed by the X-axis and Z-axis as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system.

[0017] Optionally, the length of the shee virtual slot is the distance from the nasal tip feature point, which represents the tip of the nose among the facial feature points, to the lip feature point, the upper end point of the shee virtual slot is located at a point moved 0.1 times the length from the nasal tip feature point along the positive Z-axis direction, and the lower end point of the shee virtual slot is located at a point moved 0.1 times the length from the lip feature point along the positive Z-axis direction.

[0018] Optionally, the predetermined distance is 0.25 times the height of the face, and the second preset multiplier is a numerical value of 1.

[0019] Optionally, the first preset multiple is the square root of the area of the bounding rectangle, and the preset ratio is two-thirds.

[0020] Optionally, when the control unit controls a target controllable component in a vehicle to perform a preset operation, when the target controllable component is an in-vehicle phone, rejecting an current incoming call of the in-vehicle phone; when the target controllable component is an in-vehicle multimedia device, pausing the current playback content of the in-vehicle multimedia device; when the target controllable component is a vehicle window, closing the vehicle window.

[0021] In a third aspect, embodiments of the present application provide an electronic device, the electronic device comprising a processor and a memory storing machine-executable commands executable by the processor, wherein the processor executes the machine-executable commands to implement the seat gesture detection method according to any one of the first aspect above.

[0022] In a fourth aspect, embodiments of the present application provide a machine-readable storage medium, the machine-readable storage medium storing machine-executable commands that, when invoked and executed by a processor, cause the processor to implement the seat gesture detection method according to any one of the first aspect above.

[0023] The technical solutions according to the embodiments of the present application can achieve the following beneficial effects.

[0024] In this application, after acquiring facial feature points, a first target image including the mouth region is extracted from the facial image using a constructed rectangular region to reduce interference during recognition and improve recognition efficiency. The first target image is then input into a hersing detection model to determine whether or not the facial image contains a hersing gesture. If a hersing gesture is present, the target-controllable component in the vehicle is controlled to execute a pre-set operation. This method is implemented by gesture and can complete control more quickly than conventional techniques, thus contributing to improved interaction efficiency. Furthermore, the gesture method is less susceptible to environmental influences than the voice control method, contributing to improved interference resistance in the vehicle cabin environment.

[0025] To further clarify the above-mentioned objectives, features, and advantages of this application, preferred embodiments will be described in detail below with reference to the drawings.

[0026] To more clearly explain the technical concepts of the embodiments in this application, the drawings necessary for describing the embodiments are briefly described below. The drawings described are only a selection of embodiments of this application and do not limit the scope. A person skilled in the art can obtain other relevant drawings based on these drawings without employing inventive ability. [Brief explanation of the drawing]

[0027] [Figure 1] This is a flowchart of the method for detecting a shee gesture according to an embodiment of this application. [Figure 2] This is a flowchart of another sheep gesture detection method according to an embodiment of this application. [Figure 3] This is a flowchart of another sheep gesture detection method according to an embodiment of this application. [Figure 4] This is a schematic diagram of the structure of a see gesture detection device according to an embodiment of the present application. [Figure 5] This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. [Modes for carrying out the invention]

[0028] To clarify the purpose, technical concept, and advantages of the embodiments of this application, the technical concept in the embodiments of this application will be clearly and completely described below with reference to the drawings used in the embodiments of this application, and it should be noted that the embodiments described are only a selection of embodiments of this application, not all embodiments. The components in the embodiments of this application shown using the drawings can be arranged and designed in various ways. Therefore, the following detailed description of the embodiments of this application shown in the drawings is merely to show selected embodiments of this application and does not limit the scope of the application that is to be protected. All other embodiments obtained by a person skilled in the art without using their inventive ability based on the embodiments of this application all fall within the scope of protection of this application.

[0029] Figure 1 is a flowchart of a sheep gesture detection method according to an embodiment of this application. As shown in Figure 1, this method includes the following steps.

[0030] Step 101: Based on the facial image including faces inside the vehicle captured by the in-vehicle camera, facial feature points in the facial image are obtained.

[0031] Step 102: Based on a rectangular region constructed with the target feature point in the facial feature points as the center, where the width is the value obtained by multiplying the area of ​​the bounding rectangle of the figure composed of the facial feature points by a first pre-setting multiplier, the height is the value obtained by multiplying the width by a pre-setting ratio, and the direction of extension of the face in the facial image is the direction of slope of the height, a first target image is extracted from the facial image. Here, the first pre-setting multiplier is a positive number, the pre-setting ratio is less than 1, and the target feature point belongs to the feature point of the lip region.

[0032] Step 103: Input the first target image into the hersing detection model and determine whether or not the face image contains a hersing gesture.

[0033] Step 104: If it is determined that the facial image contains the sheesh gesture, control the target controllable component in the vehicle to perform a pre-setting operation.

[0034] Specifically, an in-car camera is installed inside the vehicle, and the camera captures a real-time video of the driver's face. A facial image is obtained by extracting frames from the video. It is also possible to treat each frame in the video as a facial image. After obtaining the facial image, it is also possible to obtain facial feature points, such as mouth feature points, eyebrow feature points, eye feature points, and facial contour feature points, from that facial image.

[0035] After acquiring facial feature points, a first target image including the lip region can be extracted from the facial image based on the lip region's feature points to improve processing speed and reduce unnecessary interference. By centering the extraction on the lip region's feature points, the lip region can be included in the extraction. Then, the width is calculated by multiplying the area of ​​the bounding rectangle of the figure composed of facial feature points by a first preset multiplier, and the height is calculated by multiplying the width by the preset ratio. A rectangular region (sloping rectangle) is constructed with the direction of the face's extension in the facial image as the slope direction of the height, and the first target image is extracted from the facial image. This ensures that the extracted first target image includes not only the lip region but also the area in front of the lip region. Since a shush gesture needs to be determined by the lips, fingers, and the relationship between the two, after extracting the first target image, it is possible to determine whether or not the current facial image contains a shush gesture. If it is determined that a shush gesture is contained, the corresponding target controllable component can be controlled to perform a preset action, such as closing a car window, closing the sunroof, or turning off the sound.

[0036] This application utilizes gestures to enable control to be completed more quickly than conventional technologies, thus contributing to improved communication efficiency. Furthermore, the gesture method is less susceptible to environmental influences than voice control methods, contributing to improved interference resistance in vehicle cabin environments.

[0037] In one viable embodiment, Figure 2 is a flowchart of another shee gesture detection method according to an embodiment of the present application. As shown in Figure 2, after acquiring the facial feature points, the method further includes the following steps.

[0038] Step 201: Input the face image into the face detection model and obtain initial face feature points from the face image.

[0039] Step 202: The initial facial feature points are mapped to the same feature points in the standard three-dimensional facial model to obtain a reconstructed three-dimensional facial model.

[0040] Step 203: The three-dimensional model of the reconstructed face is mapped to the same feature points in the face image, and the target feature points are determined.

[0041] Specifically, three-dimensional feature points can provide more information about the position and orientation of the head in space, allowing the system to operate effectively even when the head rotates at various angles relative to the camera. For example, when a user is required to make a "shush" gesture with their index finger 1 cm away from their lips in a profile view, this can be more easily determined using three-dimensional facial feature points. Since in-vehicle cameras (monocular cameras) cannot directly acquire depth information in three-dimensional space, the solvePnP algorithm is used to estimate the three-dimensional coordinates corresponding to facial feature points in the face image. This process begins with detecting two-dimensional facial feature points in the input image using a pre-trained deep learning model. These two-dimensional feature points typically include prominent facial features such as the corners of the eyes, the tip of the nose, and the corners of the mouth, and their positions in the image can be accurately detected by the deep learning model. In a two-dimensional image, the origin of the coordinate system can be the upper left corner of the image, where the x-axis extends to the right, the y-axis extends downward, and the coordinate unit is pixels. The solvePnP algorithm is then used to estimate the head pose. To map these two-dimensional feature points into three-dimensional space, a standard three-dimensional face reference model, or standard three-dimensional face model, is required. The origin of the three-dimensional coordinate system corresponding to the standard three-dimensional face model is the tip of the nose, with the line connecting the two eyes pointing left as the x-axis, forward as the z-axis, and perpendicular to the x and z axes and pointing upward as the y-axis (this direction represents the direction of face extension in the face image). This reference model includes three-dimensional coordinate points corresponding to the two-dimensional feature points. Next, the pose of the face relative to the camera is estimated using the solvePnP algorithm. This algorithm can convert the two-dimensional feature points into a three-dimensional coordinate system by calculating optimal rotation and translation vectors based on the feature points in the two-dimensional image and their corresponding points in the three-dimensional model. After calculating the three-dimensional coordinates, the accuracy of the reconstruction can be verified by projecting these coordinates onto a two-dimensional plane. The accuracy of the three-dimensional coordinates can be further improved by iterative optimization algorithms such as the Levenberg-Marquardt method. If the projected two-dimensional points closely align with the feature points detected in the original image, the three-dimensional reconstruction can be considered successful. This makes it possible to reconstruct an accurate three-dimensional facial model from a typical two-dimensional image.

[0042] In one viable embodiment, Figure 3 is a flowchart of another sheep gesture detection method according to an embodiment of the present application. As shown in Figure 3, step 104 can be performed in the following steps.

[0043] Step 301: If it is determined that the face image contains the shh gesture, a second target image is extracted from the face image based on a rhombus constructed with a point located at a predetermined distance from the target feature point in the opposite direction of the extension, with the side length being the length of the face multiplied by a second preset multiplier, and the extension direction being the inclination direction.

[0044] Step 302: Based on the second target image, determine the virtual "shush" slot in the region where the index finger would be positioned when making a "shush" gesture in the second target image.

[0045] Step 303: Input the second target image into the hand feature point positioning model and obtain hand feature points.

[0046] Step 304: Based on the index finger feature point representing the index finger among the hand feature points, and the positional relationship between the index finger feature point and the shhh virtual slot, it is determined whether the shhh gesture is a correct shhh gesture.

[0047] Step 305: If the see gesture is correct, control the target controllable component in the vehicle to perform the pre-set action.

[0048] Specifically, in order to prevent errors in the shush gesture determined in step 103, for example, in misidentifying a straw, ham or sausage, or cigarette as an index finger and misidentifying it as a shush gesture, and in order to determine whether a gesture is a shush gesture, that is, whether the index finger feature points of the index finger and their positional relationship with the shush virtual slot meet the requirements of a shush gesture, the following points are specifically included.

[0049] 1. Is the index finger straight? The index finger has four key points: the fingertip, the first phalanx, the second phalanx, and the base of the third phalanx. Based on these four key points of the index finger, three bone segment vectors are calculated: the vector from the fingertip to the first phalanx, from the first phalanx to the second phalanx, and from the second phalanx to the base of the third phalanx. The angles between adjacent bone segment vectors are calculated, and a threshold (e.g., 10 degrees) is set. If all angles are less than this threshold, the index finger is determined to be straight.

[0050] 2. Is the index finger parallel to the vertical centerline of the face and within a certain distance from the front of the face? A virtual slot is projected onto the imaging plane, forming a convex hull around the four feature points of the index finger. If the area of ​​the convex hull is less than 1 / 40 of the face area, it is determined that the index finger is parallel or nearly parallel to the vertical centerline of the face.

[0051] 3. Is the index finger pointing upward? Let A be the vector from the base of the third phalanx of the index finger to the fingertip, and let B be the image plane projection vector from the lower end to the upper end of the virtual screen slot. If the angle between A and B is less than 90 degrees, it is determined that the index finger is pointing upward.

[0052] 4. Is the index finger positioned at the horizontal center of the face? Determine if the geometric centers of the four key points of the index finger lie within the region formed vertically along the face from the two ternary points of the lips.

[0053] The Shhh virtual slot is projected onto the screen plane, and the relative position of the Shhh virtual slot and the index finger is determined from the angle between the projection of the Shhh virtual slot and the two line segments of the index finger, as well as the distance between their endpoints.

[0054] To quickly complete the above detection, it is necessary to extract the lip region and the region where the index finger may be located from the face image. Specifically, a three-dimensional feature point at the center of the upper lip is obtained, moved along the negative direction of the three-dimensional y-axis by 0.25 times the head height, and projected onto the image plane to determine the center point of the rhombus region. The side length of the rhombus region is equal to one head height, and the height direction (vertical direction) of the rhombus region is parallel to the projection of the three-dimensional face feature point onto the image plane in the vertical direction. Hand feature points refer to specific important locations in the hand image, including the palm and the joints and fingertips of each finger. The output of the deep learning model includes two parts: binary classification and the coordinates of the hand keypoints. The binary classification indicates whether a hand is present in the image. For the hand keypoint coordinates, if a hand is detected, the coordinate information of each keypoint of the hand, including the position of the palm and the joints and fingertips of each finger, is output.

[0055] Based on a 3D facial model, the system calculates the ideal position for the index finger to trigger the "shush" pose. This position is called the "shush virtual slot," and the system takes the coordinates of the facial 3D feature points as input and outputs the coordinates of this virtual slot. The virtual shush slot is a 3D line segment represented by two 3D points, the coordinates of which are calculated from the facial 3D feature points. The length of the virtual shush slot is the distance from the tip of the nose to the center of the upper lip. The upper end of the virtual shush slot is located 1 / 10 of the length of the virtual shush slot along the positive Z-axis from the tip of the nose, and the lower end is located 1 / 10 of the length of the virtual shush slot along the positive Z-axis from the center of the upper lip.

[0056] After acquiring the "shush" virtual slot and the index finger feature point, it is determined whether the "shush" gesture is a correct "shush" gesture based on the positional relationship between the index finger feature point and the "shush" virtual slot.

[0057] In one viable embodiment, the target feature point is a lip feature point representing the center of the upper lip, the three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the reconstructed three-dimensional face model as the origin, the direction of the line connecting the two eyes as the X-axis, the orientation of the face as the Z-axis, and the direction perpendicular to the plane formed by the X-axis and Z-axis as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system.

[0058] In one viable embodiment, the length of the shee virtual slot is the distance from the nasal tip feature point representing the tip of the nose among the facial feature points to the lip feature point (specifically the central part of the lip), the upper end point of the shee virtual slot is located at a point moved 0.1 times the length of the shee virtual slot along the positive Z-axis from the nasal tip feature point, and the lower end point of the shee virtual slot is located at a point moved 0.1 times the length of the shee virtual slot along the positive Z-axis from the lip feature point.

[0059] In one viable embodiment, the predetermined distance is 0.25 times the height of the face, and the second preset multiplier is a numerical value of 1.

[0060] In one viable embodiment, the first preset multiplier is the square root of the area of ​​the circumscribed rectangle, and the preset ratio is two-thirds.

[0061] In one viable embodiment, the step of controlling a target-controllable component in a vehicle to perform a pre-setting operation is: If the target controllable component is an in-vehicle phone, the step of rejecting the current incoming call on the in-vehicle phone, If the target controllable component is an in-vehicle multimedia system, the steps include: pausing the current playback content of the in-vehicle multimedia system; If the target controllable component is a car window, the method includes the step of closing the car window.

[0062] Furthermore, the pre-configuration actions performed can also turn off or temporarily suspend other controllable components, such as the vehicle's air conditioning system. The specific controllable components and pre-configuration actions can be configured according to actual needs and are not limited here.

[0063] Figure 4 is a schematic diagram of the structure of a see gesture detection device according to an embodiment of this application. As shown in Figure 4, the device includes an acquisition unit 41, an extraction unit 42, a determination unit 43, and a control unit 44.

[0064] The acquisition unit 41 acquires facial feature points in the facial image, which includes faces inside the vehicle, based on the facial image captured by the in-vehicle camera.

[0065] The extraction unit 42 extracts a first target image from the face image based on a rectangular region constructed with the target feature point in the face feature points as the center, where the width is the value obtained by multiplying the area of ​​the bounding rectangle of the figure composed of the face feature points by a first pre-setting multiplier, the height is the value obtained by multiplying the width by a pre-setting ratio, and the direction of extension of the face in the face image is the direction of slope of the height. Here, the first pre-setting multiplier is a positive number, the pre-setting ratio is less than 1, and the target feature point belongs to the lip region.

[0066] The determination unit 43 inputs the first target image into the sheesh detection model and determines whether or not the face image contains a sheesh gesture.

[0067] If the control unit 44 determines that the facial image includes the sheesh gesture, it controls the target controllable component in the vehicle to perform a pre-setting operation.

[0068] In one executable embodiment, the apparatus further includes a processing unit which, after acquiring the facial feature points, inputs the facial image into a face detection model, acquires initial facial feature points in the facial image, maps the initial facial feature points to the same feature points in a standard three-dimensional face model to acquire a reconstructed three-dimensional face model, maps the reconstructed three-dimensional face model to the same feature points in the facial image to determine the target feature points.

[0069] In one executable embodiment, when the control unit determines that the face image includes the sheesh gesture, it controls a target controllable component in the vehicle to perform a pre-setting operation. If it is determined that the face image contains the "shush" gesture, a second target image is extracted from the face image based on a rhombus constructed with a point at a predetermined distance from the target feature point along the opposite direction of extension, where the side length is the length of the face multiplied by a second preset multiplier, and the extension direction is the inclination direction. Based on the second target image, determine the virtual "shush" slot in the region where the index finger is positioned when making a "shush" gesture in the second target image, The aforementioned second target image is input into the hand feature point positioning model to obtain hand feature points, Based on the index finger feature point representing the index finger among the hand feature points, and the positional relationship between the index finger feature point and the virtual shhh slot, it is determined whether the shhh gesture is a correct shhh gesture. This includes controlling a target controllable component in the vehicle to perform a pre-set action if the correct see gesture is detected.

[0070] In one viable embodiment, the target feature point is a lip feature point representing the center of the upper lip, the three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the reconstructed three-dimensional face model as the origin, the direction of the line connecting the two eyes as the X-axis, the face orientation as the Z-axis, and the direction perpendicular to the plane formed by the X-axis and Z-axis as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system.

[0071] In one viable embodiment, the length of the see virtual slot is the distance from the nasal tip feature point, which represents the nasal tip in the facial feature points, to the lip feature point, the upper end of the see virtual slot is located at a point moved 0.1 times the length from the nasal tip feature point along the positive Z-axis, and the lower end of the see virtual slot is located at a point moved 0.1 times the length from the lip feature point along the positive Z-axis.

[0072] In one viable embodiment, the predetermined distance is 0.25 times the height of the face, and the second preset multiplier is a numerical value of 1.

[0073] In one viable embodiment, the first preset multiplier is the square root of the area of ​​the circumscribed rectangle, and the preset ratio is two-thirds.

[0074] In one viable embodiment, the control unit controls a target controllable component in the vehicle to perform a pre-setting operation, If the target controllable component is an in-vehicle phone, the current incoming call on the in-vehicle phone will be rejected. When the target controllable component is an in-vehicle multimedia system, the current playback content of the in-vehicle multimedia system is paused. If the target controllable component is a car window, this includes closing the car window.

[0075] For an explanation of the principle of the sheesh gesture detection device, please refer to the explanation of the sheesh gesture detection method; a detailed explanation will be omitted here.

[0076] Figure 5 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The electronic device includes a processor 501, a storage medium 502 that stores machine-readable commands that the processor 501 can execute, and a bus 503. When the electronic device performs the see gesture detection method according to the embodiment, the processor 501 communicates with the storage medium 502 via the bus 503 and executes the machine-readable commands to perform the steps of the embodiment.

[0077] In the embodiment, the storage medium 502 is capable of executing other machine-readable commands to carry out other methods described in the embodiment, and the steps and principles of the methods to be specifically carried out can be found in the description of the embodiment, which is omitted here for further detail.

[0078] Embodiments of this application further provide a computer-readable storage medium that stores a computer program which, when executed by a processor, performs the steps shown in the above embodiment.

[0079] In the embodiments of this application, the apparatus and methods described can be implemented in other ways. The embodiments of the apparatus described above are illustrative only. For example, the unit divisions are merely logical functional divisions and may be different in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented. Furthermore, the mutual coupling, direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via several interfaces, devices or units, and may be an electrical, mechanical or other type of connection.

[0080] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in the same location or distributed across multiple network units. To achieve the objectives of this embodiment, it is possible to select some or all of the units according to actual requirements.

[0081] Furthermore, each functional unit in the embodiments of this application may be integrated into a single processing unit, function as an independent physical entity, or two or more units may be integrated into a single unit.

[0082] The aforementioned functions can be implemented in the form of a software function unit, which, when sold or used as an independent product, can be stored on a computer-readable storage medium. From this understanding, the technical proposal of this application, or any part thereof that contributes to the prior art, or any part thereof, can be implemented in the form of a software product. This computer software product is stored on a storage medium and includes a plurality of commands for a computer device (such as a personal computer, server, or network device) to execute all or part of the steps of the above method in each embodiment of this application. The storage medium includes various media capable of storing program code, such as USB disks, portable hard disks, read-only memory (ROM), random-access memory (RAM), magnetic disks, or optical disks.

[0083] Furthermore, similar symbols indicate the same thing in the drawings; therefore, if a symbol is defined in one drawing, it is unnecessary to define or interpret it further in other drawings. Also, terms such as "first," "second," and "third" should be understood as being for explanatory purposes only and not as to explicitly or implicitly indicate relative importance.

[0084] Finally, it is important to note that the above embodiments are merely specific embodiments for illustrating the technical proposal of this application and do not limit this application; therefore, the scope of protection of this application is not limited thereto. Although this application has been explained in detail using the above embodiments, it goes without saying that a person skilled in the art can propose improvements or modifications to the technical proposal described in the above embodiments within the scope of the technical scope disclosed in this application, or can make equivalent substitutions to some of the technical features therein. The essence of the technical proposals to which these improvements, modifications, or substitutions apply does not deviate from the spirit and scope of the technical proposal as exemplified in this application, and all fall within the scope of protection of this application. For this reason, the scope of protection of this application should be in accordance with the claims.

[0085] Industrial potential The sheer gesture detection method, apparatus, electronic equipment, and storage medium described in this application enable faster control completion, improved communication efficiency, reduced susceptibility to environmental influences, and enhanced interference resistance of communication in the vehicle cabin environment.

[0086] Furthermore, the sheer gesture detection method, apparatus, electronic device, and storage medium described in this application are reproducible and can be widely applied in the field of computer technology.

Claims

1. The steps include: obtaining facial feature points in a facial image, which includes faces inside the vehicle, captured by an in-vehicle camera; The steps include: extracting a first target image from the face image based on a rectangular region constructed with the target feature point among the face feature points as the center, where the width is the value obtained by multiplying the area of ​​the bounding rectangle of the figure composed of the face feature points by a first preset multiplier, the height is the value obtained by multiplying the width by a preset ratio, and the direction of extension of the face in the face image is the direction of slope of the height; The steps include inputting the first target image into a sheesh detection model and determining whether or not the face image contains a sheesh gesture, If it is determined that the facial image includes the sheesh gesture, the step of controlling a target controllable component in the vehicle to perform a pre-setting operation is included, The first pre-setting multiplier is a positive number, the pre-setting ratio is less than 1, and the target feature point belongs to the lip region. A method for detecting a sheesh gesture, characterized by the following features.

2. After obtaining the aforementioned facial feature points, The steps include inputting the aforementioned face image into a face detection model and obtaining initial face feature points in the face image, The steps include mapping the initial facial feature points to the same feature points in a standard three-dimensional facial model and obtaining a reconstructed three-dimensional facial model, The step further includes mapping the reconstructed three-dimensional face model to the same feature points in the face image and determining the target feature points. The method according to feature 1.

3. If it is determined that the facial image includes the sheesh gesture, the step of controlling a target-controllable component in the vehicle to perform a pre-setting operation is: If it is determined that the face image contains the "shush" gesture, the steps include: extracting a second target image from the face image based on a rhombus constructed with a point at a predetermined distance from the target feature point along the opposite direction of extension, with the side length being the length of the face multiplied by a second preset multiplier, and the extension direction being the inclination direction; Based on the second target image, the steps include determining a virtual "shush" slot in the region where the index finger is positioned when making a "shush" gesture in the second target image, The steps include inputting the aforementioned second target image into the hand feature point positioning model and obtaining hand feature points, The steps include determining whether the "shush" gesture is a correct "shush" gesture based on the index finger feature point representing the index finger among the hand feature points, and the positional relationship between the index finger feature point and the "shush" virtual slot, The steps include controlling a target controllable component in the vehicle to perform a pre-set action if the see gesture is correct. The method according to feature 1.

4. The aforementioned target feature point is a lip feature point representing the center of the upper lip, and the three-dimensional coordinate system is constructed with the nose tip feature point representing the tip of the nose in the three-dimensional face reconstruction model as the origin, the direction of the line connecting both eyes as the X-axis, the face orientation as the Z-axis, and the direction perpendicular to the plane formed by the X-axis and Z-axis as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system. The method according to feature 3.

5. The length of the virtual sheathing slot is the distance from the nasal tip feature point, which represents the tip of the nose among the facial feature points, to the lip feature point. The upper end point of the virtual sheathing slot is located at a point moved 0.1 times the length from the nasal tip feature point along the positive Z-axis direction, and the lower end point of the virtual sheathing slot is located at a point moved 0.1 times the length from the lip feature point along the positive Z-axis direction. The method according to feature 4.

6. The predetermined distance is 0.25 times the height of the face, and the second pre-setting multiplier is a value of 1. The method according to feature 3.

7. The first pre-setting multiplier is the square root of the area of ​​the circumscribed rectangle, and the pre-setting ratio is two-thirds. The method according to feature 1.

8. The step of controlling a target-controllable component in the vehicle to perform a pre-set operation is: If the target controllable component is an in-vehicle phone, the step of rejecting the current incoming call on the in-vehicle phone, If the target controllable component is an in-vehicle multimedia system, the steps include: pausing the current playback content of the in-vehicle multimedia system; If the target controllable component is a car window, the step includes closing the car window. The method according to feature 1.

9. An acquisition unit that acquires facial feature points in a facial image, including faces inside the vehicle, based on a facial image captured by an in-vehicle camera, An extraction unit extracts a first target image from a face image based on a rectangular region constructed with a target feature point among the face feature points as the center, where the width is the value obtained by multiplying the area of ​​the bounding rectangle of the figure composed of the face feature points by a first preset multiplier, the height is the value obtained by multiplying the width by a preset ratio, and the direction of extension of the face in the face image is the direction of slope of the height. A determination unit inputs the first target image into a shhh detection model and determines whether or not the face image contains a shhh gesture, A control unit that controls a target-controllable component in the vehicle to perform a pre-setting operation when it is determined that the facial image includes the sheesh gesture, In the extraction unit, the first pre-setting multiplier is a positive number, the pre-setting ratio is less than 1, and the target feature point belongs to the feature point of the lip region. A device for detecting a sheep gesture, characterized by the following features.

10. The system includes a processor and a memory that stores machine-executable commands that the processor can execute, and the processor executes the machine-executable commands to realize the see gesture detection method according to any one of claims 1 to 8. An electronic device characterized by the following features.

11. When called and executed by the processor, the processor stores a machine-executable command that enables the detection method for see gestures described in any one of claims 1 to 8. A machine-readable storage medium characterized by the following features.

Images