Shushing gesture detection method and apparatus, electronic device, and storage medium

Abstract

The present application provides a shushing gesture detection method and apparatus, an electronic device, and a storage medium. In the present application, using a gesture to control a target controllable component within a vehicle allows for faster control, helping to improve interaction efficiency. Moreover, gesture control is less affected by the environment than voice control, helping to improve anti-interference capability during interactions in vehicle cabin environments.

Description

A method, apparatus, electronic device, and storage medium for detecting a shushing gesture.

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411170859.7, filed on August 26, 2024, entitled "A method, apparatus, electronic device and storage medium for detecting a shushing gesture", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and more specifically, to a method, apparatus, electronic device, and storage medium for detecting a shushing gesture. Background Technology

[0004] Currently, some functions in vehicles are interacted with via control buttons (including physical or touch buttons), such as pausing music or closing the sunroof. Users need to use the corresponding buttons to control these functions, which requires locating the correct button. Although voice control systems exist as a supplement, the accuracy and response speed of voice recognition decrease significantly when there is noise in the cabin. Therefore, there is an urgent need for an interaction method that is highly resistant to interference and convenient to use.

[0005] Summary of the Invention

[0006] In view of this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for detecting a shushing gesture, so as to improve the anti-interference capability when interacting in a vehicle cabin environment and enable efficient interaction.

[0007] In a first aspect, embodiments of this application provide a method for detecting a shushing gesture, the method comprising:

[0008] Based on the facial images of people inside the vehicle captured by the vehicle-mounted camera, obtain the facial feature points in the facial images;

[0009] A first target image is extracted from a face image based on a rectangular region constructed with the target feature point as the center, the area of ​​the bounding rectangle formed by the face feature points as a first preset multiple as the width, the width as a preset ratio as the height, and the extension direction of the face in the face image as the tilt direction of the height. Here, the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to the lip region.

[0010] The first target image is input into the shushing detection model to determine whether the face image contains a shushing gesture;

[0011] When it is determined that the facial image contains the shushing gesture, the target controllable component in the vehicle is controlled to perform a preset action.

[0012] Optionally, after obtaining the facial feature points, the method further includes:

[0013] The face image is input into the face detection model to obtain the initial face feature points in the face image;

[0014] The initial facial feature points are mapped onto the same feature points in a standard 3D facial model to obtain a reconstructed 3D facial model.

[0015] The reconstructed 3D model of the face is mapped back to the same feature point in the face image to determine the target feature point.

[0016] Optionally, when it is determined that the facial image contains the shushing gesture, controlling the target controllable component in the vehicle to perform a preset action includes:

[0017] When it is determined that the face image contains the shushing gesture, a second target image is extracted from the face image based on a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extension direction as the center, a second preset multiple of the face as the side length, and the extension direction as the tilt direction.

[0018] Based on the second target image, determine the virtual groove for the shushing sound in the area where the index finger is located when making the shushing sound in the second target image;

[0019] The second target image is input into the hand feature point localization model to obtain hand feature points;

[0020] Based on the index finger feature point used to represent the index finger among the hand feature points, and its positional relationship with the virtual shushing slot, it is determined whether the shushing gesture is a correct shushing gesture.

[0021] If the shushing gesture is correct, the target controllable component in the vehicle will perform a preset action.

[0022] Optionally, the target feature point is a lip feature point used to represent the center of the upper lip. The three-dimensional coordinate system is constructed with the nasal tip feature point representing the nasal tip in the face reconstruction three-dimensional model as the origin, the direction of the line connecting the eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane containing the X-axis and Z-axis as the Y-axis. The extension direction is the Y-axis direction of the three-dimensional coordinate system.

[0023] Optionally, the length of the virtual shushing groove is the distance from the nose tip feature point (used to represent the nose tip) to the lip feature point in the facial feature points. The upper end point of the virtual shushing groove is located at 0.1 times the length of the nose tip feature point along the positive Z-axis, and the lower end point of the virtual shushing groove is located at 0.1 times the length of the lip feature point along the positive Z-axis.

[0024] Optionally, the specified distance is 0.25 times the face height, and the second preset multiple is a value of 1.

[0025] Optionally, the first preset multiple is the square root of the area of ​​the circumscribed rectangle, and the preset ratio is two-thirds.

[0026] Optionally, the controllable component in the vehicle performs a preset action, including:

[0027] When the target controllable component is a vehicle phone, the incoming call to the vehicle phone is rejected.

[0028] When the target controllable component is an in-vehicle multimedia system, the currently playing content of the in-vehicle multimedia system is paused.

[0029] When the target controllable component is a car window, then the car window is closed.

[0030] Secondly, embodiments of this application provide a shushing gesture detection device, the device comprising:

[0031] The acquisition unit is used to acquire facial feature points in a facial image, including a face, captured by an in-vehicle camera inside a vehicle.

[0032] The cropping unit is used to crop a first target image from a face image based on a rectangular area constructed with the target feature point in the face feature point as the center, the area of ​​the outer rectangle of the shape formed by the face feature points as a first preset multiple as the width, the width as a preset ratio as the height, and the extension direction of the face in the face image as the tilt direction of the height. The first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to the feature point of the lip region.

[0033] The judgment unit is used to input the first target image into the shushing detection model and determine whether the face image contains a shushing gesture;

[0034] The control unit is configured to control a target controllable component in the vehicle to perform a preset action when it is determined that the facial image contains the shushing gesture.

[0035] Optionally, the device further includes:

[0036] The processing unit is configured to, after obtaining the facial feature points, input the facial image into a facial detection model to obtain initial facial feature points in the facial image; and to map the initial facial feature points onto the same feature point in a standard 3D facial model to obtain a reconstructed 3D facial model; and to map the reconstructed 3D facial model back onto the same feature point in the facial image to determine the target feature point.

[0037] Optionally, when the control unit determines that the facial image contains the shushing gesture, when controlling a target controllable component in the vehicle to perform a preset action, the following steps are included:

[0038] When it is determined that the face image contains the shushing gesture, a second target image is extracted from the face image based on a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extension direction as the center, a second preset multiple of the face as the side length, and the extension direction as the tilt direction.

[0039] Based on the second target image, determine the virtual groove for the shushing sound in the area where the index finger is located when making the shushing sound in the second target image;

[0040] The second target image is input into the hand feature point localization model to obtain hand feature points;

[0041] Based on the index finger feature point used to represent the index finger among the hand feature points, and its positional relationship with the virtual shushing slot, it is determined whether the shushing gesture is a correct shushing gesture.

[0042] If the shushing gesture is correct, the target controllable component in the vehicle will perform a preset action.

[0043] Optionally, the target feature point is a lip feature point used to represent the center of the upper lip. The three-dimensional coordinate system is constructed with the nasal tip feature point representing the nasal tip in the face reconstruction three-dimensional model as the origin, the direction of the line connecting the eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane containing the X-axis and Z-axis as the Y-axis. The extension direction is the Y-axis direction of the three-dimensional coordinate system.

[0044] Optionally, the length of the virtual shushing groove is the distance from the nose tip feature point (used to represent the nose tip) to the lip feature point in the facial feature points. The upper end point of the virtual shushing groove is located at 0.1 times the length of the nose tip feature point along the positive Z-axis, and the lower end point of the virtual shushing groove is located at 0.1 times the length of the lip feature point along the positive Z-axis.

[0045] Optionally, the specified distance is 0.25 times the face height, and the second preset multiple is a value of 1.

[0046] Optionally, the first preset multiple is the square root of the area of ​​the circumscribed rectangle, and the preset ratio is two-thirds.

[0047] Optionally, when the control unit is used to control a target controllable component in the vehicle to perform a preset action, it includes:

[0048] When the target controllable component is a vehicle phone, the incoming call to the vehicle phone is rejected.

[0049] When the target controllable component is an in-vehicle multimedia system, the currently playing content of the in-vehicle multimedia system is paused.

[0050] When the target controllable component is a car window, then the car window is closed.

[0051] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the shushing gesture detection method described in any one of the first aspects above.

[0052] Fourthly, embodiments of this application provide a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to implement the shushing gesture detection method described in any one of the first aspects above.

[0053] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0054] In this application, after obtaining facial feature points, in order to reduce interference during recognition and improve recognition efficiency, a first target image containing the mouth region is extracted from the facial image using a constructed rectangular region. Then, the first target image is input into a shushing detection model to determine whether the facial image contains a shushing gesture. If it contains a shushing finger, the target controllable component in the vehicle is controlled to perform a preset action. The above method is accomplished through gestures, which can complete the control more quickly than the prior art, thus improving the interaction efficiency. Moreover, since gestures are less affected by the environment than voice control, they are beneficial to improving the anti-interference capability when interacting in the vehicle cabin environment.

[0055] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 is a flowchart illustrating a shushing gesture detection method provided in an embodiment of this application;

[0058] Figure 2 is a flowchart illustrating another method for detecting a shushing gesture provided in an embodiment of this application;

[0059] Figure 3 is a flowchart illustrating another method for detecting a shushing gesture provided in an embodiment of this application;

[0060] Figure 4 is a schematic diagram of a shushing gesture detection device provided in an embodiment of this application;

[0061] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0063] Figure 1 is a flowchart illustrating a shushing gesture detection method provided in an embodiment of this application. As shown in Figure 1, the method includes the following steps:

[0064] Step 101: Based on the face image inside the vehicle captured by the vehicle-mounted camera, obtain the face feature points in the face image.

[0065] Step 102: Based on a rectangular area constructed with the target feature point in the facial feature points as the center, the area of ​​the outer rectangle of the graphic formed by the facial feature points as a first preset multiple as the width, the width as a preset ratio as the height, and the extension direction of the face in the facial image as the tilt direction of the height, a first target image is extracted from the facial image. Here, the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to the feature point of the lip region.

[0066] Step 103: Input the first target image into the shushing detection model to determine whether the face image contains a shushing gesture.

[0067] Step 104: When it is determined that the face image contains the shushing gesture, control the target controllable component in the vehicle to perform a preset action.

[0068] Specifically, an in-vehicle camera is installed in the cabin to capture real-time video of the driver's face. Frames are extracted from the video to obtain the facial image. Alternatively, each frame of the video can be used as a facial image. After obtaining the facial image, facial feature points are extracted, such as mouth feature points, eyebrow feature points, eye feature points, and facial contour feature points.

[0069] After obtaining facial feature points, to improve processing speed and reduce unnecessary interference, a first target image including the lip region can be cropped from the face image based on the feature points of the lip region. The cropping is centered on the feature points of the lip region, thus including the lip region. Then, a rectangular area (tilted rectangle) is constructed with a width that is a first preset multiple of the area of ​​the bounding rectangle formed by the facial feature points, a height that is a preset ratio of the width, and a tilted direction that is the extension direction of the face in the face image as the height. The first target image is then cropped from the face image. This ensures that the cropped first target image includes not only the lip region but also the area in front of the lip region. Since the shushing gesture needs to be judged by the lips, fingers, and the relationship between the two, after the first target image is cropped, it can be determined whether the current face image contains the shushing gesture. When the shushing gesture is determined to be contained, the corresponding controllable component can be controlled to perform a preset action, such as closing the car window, closing the sunroof, or turning off the sound.

[0070] This application uses gestures to complete the operation, which allows for faster control compared to existing technologies. This improves interaction efficiency. Furthermore, gestures are less affected by the environment than voice control, which enhances the anti-interference capability when interacting in a vehicle cabin environment.

[0071] In a feasible implementation, Figure 2 is a flowchart illustrating another shushing gesture detection method provided by an embodiment of this application. As shown in Figure 2, after obtaining the facial feature points, the method further includes the following steps:

[0072] Step 201: Input the face image into the face detection model to obtain the initial face feature points in the face image.

[0073] Step 202: Map the initial facial feature points to the same feature points in the standard 3D facial model to obtain the reconstructed 3D facial model.

[0074] Step 203: Map the reconstructed 3D face model back to the same feature point in the face image to determine the target feature point.

[0075] Specifically, because 3D feature points provide more information about the head's position and orientation in space, the system can work effectively even when the head has different rotation angles relative to the camera. For example, when asking someone to make a shushing gesture with their index finger 1cm from their lips, 3D facial feature points make this easier to determine. Since the vehicle-mounted camera (a monocular camera) cannot directly capture depth information in 3D space, the solvePnP algorithm is used to estimate the 3D coordinates of facial feature points in the face image. This process begins by using a pre-trained deep learning model to detect 2D facial feature points in the input image. These 2D feature points typically include prominent facial features such as the corners of the eyes, the tip of the nose, and the corners of the mouth. Their positions in the image can be accurately detected by the deep learning model. In the 2D image, the origin of the coordinate system can be the upper left corner of the image, where the x-axis extends to the right and the y-axis extends downwards, with the coordinate unit being pixels. The solvePnP algorithm is used for head pose estimation. To map these 2D feature points to 3D space, a standard 3D face reference model is needed: a standard 3D face model. The origin of the 3D coordinate system corresponding to this model is the tip of the nose, the x-axis is to the left of the line connecting the eyes, the z-axis is forward, and the y-axis is perpendicular to both the x and z axes and extends upwards (this direction represents the extension of the face in the image). This reference model contains the 3D coordinates corresponding to the 2D feature points. Then, we use the solvePnP algorithm to estimate the face's pose relative to the camera. This algorithm calculates the optimal rotation and translation vectors based on the feature points in the 2D image and their corresponding points in the 3D model, thus transforming the 2D feature points into the 3D coordinate system. After calculating the 3D coordinates, the accuracy of the reconstruction can be verified by projecting these coordinates back onto the 2D plane. Iterative optimization algorithms, such as the Levenberg-Marquardt method, are used to further improve the accuracy of the 3D coordinates. If the projected 2D points are closely aligned with the feature points detected in the original image, then the 3D reconstruction is considered successful. In this way, an accurate 3D face model can be reconstructed from a regular 2D image.

[0076] In a feasible implementation, Figure 3 is a flowchart illustrating another shushing gesture detection method provided by an embodiment of this application. As shown in Figure 3, step 104 can be achieved through the following steps:

[0077] Step 301: When it is determined that the face image contains the shushing gesture, a second target image is extracted from the face image based on a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extension direction as the center, a second preset multiple of the face as the side length, and the extension direction as the tilt direction.

[0078] Step 302: Based on the second target image, determine the virtual groove for the shushing sound in the area where the index finger is located when making the shushing sound in the second target image.

[0079] Step 303: Input the second target image into the hand feature point model to obtain hand feature points.

[0080] Step 304: Based on the index finger feature point used to represent the index finger among the hand feature points, and its positional relationship with the virtual shushing slot, determine whether the shushing gesture is a correct shushing gesture.

[0081] Step 305: If the shushing gesture is correct, control the target controllable component in the vehicle to perform the preset action.

[0082] Specifically, to prevent errors in the shushing gesture judgment in step 103, such as misidentifying a straw, sausage, or cigarette as an index finger and thus misjudging it as a shushing gesture, and to ensure that the gesture conforms to the shushing gesture, specifically whether the index finger's characteristic points and their positional relationship with the shushing virtual slot meet the requirements of the shushing gesture, the following points are included:

[0083] 1. Is the index finger straight? The index finger consists of four key points: the fingertip, the first phalanx, the second phalanx, and the base of the third phalanx. Based on these four key points, three bone segment vectors are calculated: the vector from the fingertip to the first phalanx, the vector from the first phalanx to the second phalanx, and the vector from the second phalanx to the base of the third phalanx. The angles between adjacent bone segment vectors are calculated, and a threshold (e.g., 10 degrees) is set. If all angles are less than this threshold, the index finger is considered straight.

[0084] 2. Whether the index finger is parallel to the vertical midline of the face and within a certain distance in front of the face. Project the virtual groove of the squeegee onto the imaging plane, forming a convex hull with the four feature points of the index finger. When the area of ​​the convex hull is less than 1 / 40 of the face area, the index finger is nearly parallel to the vertical midline of the face.

[0085] 3. Is the index finger pointing upwards? Let A be the vector from the base of the third phalanx of the index finger to the fingertip, and B be the projection vector from the lower end to the upper end of the virtual groove on the image plane. When the angle between A and B is less than 90 degrees, the index finger is pointing upwards.

[0086] 4. Is the index finger located at the horizontal center of the face? The geometric center of the four key points of the index finger lies between the two third points of the lips, forming the area along the numerical direction of the face.

[0087] The relative position of the virtual squeegee and the index finger is determined by projecting the virtual squeegee onto the screen plane and measuring the angle between the projection of the virtual squeegee and the two line segments of the index finger, as well as the distance between their endpoints.

[0088] To quickly complete the above detection, it is necessary to crop the facial image, including the lip region and the area where the index finger might be present. Specifically, take the 3D feature point at the center of the upper lip and move it 0.25 head heights along the negative y-axis. Project this onto the image plane to determine the center point of the rhombus region. The side length of the rhombus region is equal to the height of a head, and the height direction (vertical direction) of the rhombus region is parallel to the vertical projection of the 3D facial feature point onto the image plane. Hand feature points refer to specific key locations in the hand image, including the palm and the joints and fingertips of each finger. The output of the deep learning model includes two parts: binary classification, indicating whether a hand is present in the image; and hand keypoint coordinates. If a hand is detected, the coordinate information of each keypoint of the hand will be output, specifically including the palm and the joints and fingertips of all fingers.

[0089] Based on a 3D facial model, the ideal position for the index finger to trigger a shushing sound is calculated; this position is called the "shushing virtual slot." The input is the coordinates of the 3D feature points of the face, and the output is the coordinates of the shushing virtual slot. The shushing virtual slot is a 3D line segment represented by two 3D points, whose coordinates are calculated from the 3D feature points of the face. The length of the shushing virtual slot is the distance from the tip of the nose to the midpoint of the upper lip. The upper endpoint of the shushing virtual slot is located at the tip of the nose, moved along the positive z-axis by 1 / 10 of the shushing virtual slot length, and the lower endpoint is located at the midpoint of the upper lip, moved along the positive z-axis by 1 / 10 of the shushing virtual slot length.

[0090] After obtaining the virtual groove for the shushing sound and the feature point of the index finger, the system determines whether the shushing gesture is the correct shushing gesture based on the feature point of the index finger and its positional relationship with the virtual groove for the shushing sound.

[0091] In a feasible implementation, the target feature point is a lip feature point used to represent the center of the upper lip. The three-dimensional coordinate system is constructed with the nasal tip feature point representing the nasal tip in the face reconstruction three-dimensional model as the origin, the direction of the line connecting the eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane containing the X-axis and Z-axis as the Y-axis. The extension direction is the Y-axis direction of the three-dimensional coordinate system.

[0092] In one feasible implementation, the length of the virtual shushing groove is the distance from the nose tip feature point (which represents the nose tip) to the lip feature point (specifically, the middle of the lip) among the facial feature points. The upper end point of the virtual shushing groove is located at 0.1 times the length of the virtual shushing groove along the positive Z-axis of the nose tip feature point, and the lower end point of the virtual shushing groove is located at 0.1 times the length of the virtual shushing groove along the positive Z-axis of the lip feature point.

[0093] In one feasible implementation, the specified distance is 0.25 times the face height, and the second preset multiple is a value of 1.

[0094] In one feasible implementation, the first preset multiple is the square root of the area of ​​the circumscribed rectangle, and the preset ratio is two-thirds.

[0095] In one feasible implementation, the controllable component in the vehicle performs a preset action, including:

[0096] When the target controllable component is a vehicle phone, the incoming call to the vehicle phone is rejected.

[0097] When the target controllable component is an in-vehicle multimedia system, the currently playing content of the in-vehicle multimedia system is paused.

[0098] When the target controllable component is a car window, then the car window is closed.

[0099] It should be noted that the preset actions can also shut down or temporarily disable other controllable components, such as the vehicle's air conditioning. The specific controllable components and preset actions can be set according to actual needs, and no specific limitations are made here.

[0100] Figure 4 is a schematic diagram of a shushing gesture detection device provided in an embodiment of this application. As shown in Figure 4, the device includes:

[0101] The acquisition unit 41 is used to acquire facial feature points in the facial image captured by the vehicle camera inside the vehicle, including a human face.

[0102] The cropping unit 42 is used to crop a first target image from the face image based on a rectangular area constructed with the target feature point in the face feature point as the center, the area of ​​the outer rectangle of the graphic formed by the face feature points as the width, the width as a preset ratio as the height, and the extension direction of the face in the face image as the tilt direction of the height. The first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to the feature point of the lip region.

[0103] Judgment unit 43 is used to input the first target image into the shushing detection model and determine whether the face image contains a shushing gesture;

[0104] Control unit 44 is used to control a target controllable component in the vehicle to perform a preset action when it is determined that the face image contains the shushing gesture.

[0105] In one feasible implementation, the device further includes:

[0106] The processing unit is configured to, after obtaining the facial feature points, input the facial image into a facial detection model to obtain initial facial feature points in the facial image; and to map the initial facial feature points onto the same feature point in a standard 3D facial model to obtain a reconstructed 3D facial model; and to map the reconstructed 3D facial model back onto the same feature point in the facial image to determine the target feature point.

[0107] In one feasible implementation, the control unit is configured to control a target controllable component in the vehicle to perform a preset action when it is determined that the facial image contains the shushing gesture, including:

[0108] When it is determined that the face image contains the shushing gesture, a second target image is extracted from the face image based on a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extension direction as the center, a second preset multiple of the face as the side length, and the extension direction as the tilt direction.

[0109] Based on the second target image, determine the virtual groove for the shushing sound in the area where the index finger is located when making the shushing sound in the second target image;

[0110] The second target image is input into the hand feature point localization model to obtain hand feature points;

[0111] Based on the index finger feature point used to represent the index finger among the hand feature points, and its positional relationship with the virtual shushing slot, it is determined whether the shushing gesture is a correct shushing gesture.

[0112] If the shushing gesture is correct, the target controllable component in the vehicle will perform a preset action.

[0113] In a feasible implementation, the target feature point is a lip feature point used to represent the center of the upper lip. The three-dimensional coordinate system is constructed with the nasal tip feature point representing the nasal tip in the face reconstruction three-dimensional model as the origin, the direction of the line connecting the eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane containing the X-axis and Z-axis as the Y-axis. The extension direction is the Y-axis direction of the three-dimensional coordinate system.

[0114] In one feasible implementation, the length of the virtual shushing groove is the distance from the nose tip feature point (used to represent the nose tip) to the lip feature point in the facial feature points, the upper end point of the virtual shushing groove is located at 0.1 times the length of the nose tip feature point along the positive Z-axis, and the lower end point of the virtual shushing groove is located at 0.1 times the length of the lip feature point along the positive Z-axis.

[0115] In one feasible implementation, the specified distance is 0.25 times the face height, and the second preset multiple is a value of 1.

[0116] In one feasible implementation, the first preset multiple is the square root of the area of ​​the circumscribed rectangle, and the preset ratio is two-thirds.

[0117] In one feasible implementation, when the control unit is used to control a target controllable component in the vehicle to perform a preset action, it includes:

[0118] When the target controllable component is a vehicle phone, the incoming call to the vehicle phone is rejected.

[0119] When the target controllable component is an in-vehicle multimedia system, the currently playing content of the in-vehicle multimedia system is paused.

[0120] When the target controllable component is a car window, then the car window is closed.

[0121] For an explanation of the principle of the shushing gesture detection device, please refer to the relevant explanation of the shushing gesture detection method, which will not be explained in detail here.

[0122] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including: a processor 501, a storage medium 502 and a bus 503. The storage medium 502 stores machine-readable instructions that can be executed by the processor 501. When the electronic device runs a shushing gesture detection method as described in the embodiment, the processor 501 and the storage medium 502 communicate through the bus 503. The processor 501 executes the machine-readable instructions to perform the steps as described in the embodiment.

[0123] In this embodiment, the storage medium 502 may also execute other machine-readable instructions to perform other methods as described in the embodiment. For details on the specific execution steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0124] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps shown in the above embodiments.

[0125] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0126] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0128] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0130] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims. Industrial applicability

[0131] This application provides a method, device, electronic device, and storage medium for detecting a shushing gesture, which enables faster control, improves interaction efficiency, is less affected by the environment, and enhances anti-interference capabilities when interacting in a vehicle cabin environment.

[0132] Furthermore, it is understood that the shushing gesture detection method, apparatus, electronic device, and storage medium of this application are reproducible and can be widely applied in the field of computer technology.

Claims

1. A method for detecting a shushing gesture, characterized in that, The method includes: Based on the facial images of people inside the vehicle captured by the vehicle-mounted camera, obtain the facial feature points in the facial images; A first target image is extracted from a face image based on a rectangular region constructed with the target feature point as the center, the area of ​​the bounding rectangle formed by the face feature points as a first preset multiple as the width, the width as a preset ratio as the height, and the extension direction of the face in the face image as the tilt direction of the height. Here, the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to the lip region. The first target image is input into the shushing detection model to determine whether the face image contains a shushing gesture; When it is determined that the facial image contains the shushing gesture, the target controllable component in the vehicle is controlled to perform a preset action.

2. The method as described in claim 1, characterized in that, After obtaining the facial feature points, the method further includes: The face image is input into the face detection model to obtain the initial face feature points in the face image; The initial facial feature points are mapped onto the same feature points in a standard 3D facial model to obtain a reconstructed 3D facial model. The reconstructed 3D model of the face is mapped back to the same feature point in the face image to determine the target feature point.

3. The method as described in claim 1, characterized in that, When it is determined that the facial image contains the shushing gesture, the controllable component in the vehicle is controlled to perform a preset action, including: When it is determined that the face image contains the shushing gesture, a second target image is extracted from the face image based on a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extension direction as the center, a second preset multiple of the face as the side length, and the extension direction as the tilt direction. Based on the second target image, determine the virtual groove for the shushing sound in the area where the index finger is located when making the shushing sound in the second target image; The second target image is input into the hand feature point localization model to obtain hand feature points; Based on the index finger feature point used to represent the index finger among the hand feature points, and its positional relationship with the virtual shushing slot, it is determined whether the shushing gesture is a correct shushing gesture. If the shushing gesture is correct, the target controllable component in the vehicle will perform a preset action.

4. The method as described in claim 3, characterized in that, The target feature point is the lip feature point used to represent the center of the upper lip. The three-dimensional coordinate system is constructed with the nasal tip feature point representing the tip of the nose in the face reconstruction three-dimensional model as the origin, the direction of the line connecting the eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane containing the X-axis and Z-axis as the Y-axis. The extension direction is the Y-axis direction of the three-dimensional coordinate system.

5. The method as described in claim 4, characterized in that, The length of the virtual groove for the shushing sound is the distance from the nose tip feature point (used to represent the nose tip) to the lip feature point. The upper end point of the virtual groove for the shushing sound is located at 0.1 times the length of the nose tip feature point along the positive Z-axis, and the lower end point of the virtual groove for the shushing sound is located at 0.1 times the length of the lip feature point along the positive Z-axis.

6. The method as described in claim 3, characterized in that, The specified distance is 0.25 times the height of the face, and the second preset multiple is 1.

7. The method as described in claim 1, characterized in that, The first preset multiple is the square root of the area of ​​the circumscribed rectangle, and the preset ratio is two-thirds.

8. The method as described in claim 1, characterized in that, The controllable components in the vehicle are said to perform preset actions, including: When the target controllable component is a vehicle phone, the incoming call to the vehicle phone is rejected. When the target controllable component is an in-vehicle multimedia system, the currently playing content of the in-vehicle multimedia system is paused. When the target controllable component is a car window, then the car window is closed.

9. A shushing gesture detection device, characterized in that, The device includes: The acquisition unit is used to acquire facial feature points in a facial image, including a face, captured by an in-vehicle camera inside a vehicle. The cropping unit is used to crop a first target image from a face image based on a rectangular area constructed with the target feature point in the face feature point as the center, the area of ​​the outer rectangle of the shape formed by the face feature points as a first preset multiple as the width, the width as a preset ratio as the height, and the extension direction of the face in the face image as the tilt direction of the height. The first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to the feature point of the lip region. The judgment unit is used to input the first target image into the shushing detection model and determine whether the face image contains a shushing gesture; The control unit is configured to control a target controllable component in the vehicle to perform a preset action when it is determined that the facial image contains the shushing gesture.

10. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the shushing gesture detection method according to any one of claims 1-8.

11. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the shushing gesture detection method according to any one of claims 1-8.

Images