Method, device, electronic device and storage medium for detecting hushing gesture
Through the on-board camera and face feature point processing technology, the boo gestures are recognized and executed, which solves the problem of difficulty in identifying interactive commands in the cabin environment, and realizes an efficient and anti-interference gesture control method.
Patent Information
- Application Number
- CN202411170859.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-08-26
AI Technical Summary
In the cabin environment, it is difficult for the prior art to effectively identify and execute interactive commands, especially under noise interference, the accuracy and speed of voice control are significantly reduced, and the interaction method with anti-interference ability is lacking.
The face image is obtained through the on-board camera, the face feature points are extracted, the target image is intercepted by building a rectangular area, and input it into the boo detection model to determine whether it contains boo gestures, and control the vehicle's controllable components to perform preset actions, using three-dimensional faces Model and hand feature points accurately define boo gestures.
It improves the interaction efficiency and anti-interference ability in the cabin environment, and realizes the rapid and accurate control of the vehicle function through gestures, which is less affected by the environment than voice control.
Smart Images

Figure CN118675204B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for detecting a shushing gesture. Background Art
[0002] At present, the interaction method of some functions in the vehicle is carried out through control buttons (including physical buttons or touch buttons), for example: pausing music, closing the sunroof, etc. When the user controls the above functions, he needs to use the corresponding buttons to achieve it. This control method requires finding the corresponding button before completing the control. Although there is a voice control system as a supplement, when there is noise in the cabin, the accuracy and response speed of voice recognition will be significantly reduced. Therefore, there is an urgent need for an interaction method with strong anti-interference ability and convenient interaction. Summary of the invention
[0003] In view of this, embodiments of the present application provide a method, device, electronic device and storage medium for detecting a hushing gesture, so as to improve the anti-interference capability during interaction in a vehicle cabin environment and enable efficient interaction.
[0004] In a first aspect, an embodiment of the present application provides a method for detecting a shushing gesture, the method comprising:
[0005] According to a facial image including a human face inside a vehicle captured by a vehicle-mounted camera, acquiring facial feature points in the facial image;
[0006] A first target image is intercepted from the face image according to a rectangular area constructed with a target feature point among the face feature points as the center, a first preset multiple of the area of a circumscribed rectangle of a figure formed by the face feature points as the width, a preset ratio of the width as the height, and an extension direction of the face in the face image as the inclination direction of the height, wherein the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to a feature point of the lip area;
[0007] Inputting the first target image into a shushing detection model to determine whether the face image contains a shushing gesture;
[0008] When it is determined that the face image contains the shushing gesture, the target controllable component in the vehicle is controlled to perform a preset action.
[0009] Optionally, after obtaining the facial feature points, the method further includes:
[0010] Inputting the face image into a face detection model to obtain initial face feature points in the face image;
[0011] Mapping the initial facial feature points to the same feature points in a standard three-dimensional face model to obtain a reconstructed three-dimensional face model;
[0012] The reconstructed three-dimensional face model is mapped back to the same feature point in the face image to determine the target feature point.
[0013] Optionally, when it is determined that the face image contains the shushing gesture, controlling a target controllable component in the vehicle to perform a preset action includes:
[0014] When it is determined that the face image contains the shushing gesture, a second target image is intercepted from the face image according to a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extending direction as the center, a second preset multiple of the face as the side length, and the extending direction as the inclined direction;
[0015] determining, according to the second target image, a shushing virtual slot in a region where the index finger is located when shushing is performed in the second target image;
[0016] Inputting the second target image into a positioning hand feature point model to obtain hand feature points;
[0017] determining whether the shushing gesture is a correct shushing gesture according to an index finger feature point among the hand feature points used to represent the index finger and a positional relationship with the shushing virtual slot;
[0018] If the shush gesture is correct, the target controllable component in the vehicle is controlled to perform a preset action.
[0019] Optionally, the target feature point is a lip feature point used to represent the center of the upper lip, and the three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the reconstructed three-dimensional model of the face as the origin, the direction of the line connecting the two eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane where the X-axis and the Z-axis are located as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system.
[0020] Optionally, the length of the virtual hushing groove is the distance from a nose tip feature point used to represent the nose tip among facial feature points to the lip feature point, the upper endpoint of the virtual hushing groove is located at 0.1 times the length of the nose tip feature along the positive direction of the Z axis, and the lower endpoint of the virtual hushing groove is located at 0.1 times the length of the lip feature point along the positive direction of the Z axis.
[0021] Optionally, the designated distance is 0.25 times the height of the face, and the second preset multiple is a value of 1.
[0022] Optionally, the first preset multiple is the square root of the area of the circumscribed rectangle, and the preset ratio is two-thirds.
[0023] Optionally, controlling a target controllable component in the vehicle to perform a preset action includes:
[0024] When the target controllable component is a car phone, rejecting the current incoming call of the car phone;
[0025] When the target controllable component is an in-vehicle multimedia, pausing the content currently being played by the in-vehicle multimedia;
[0026] When the target controllable component is a vehicle window, the vehicle window is closed.
[0027] In a second aspect, an embodiment of the present application provides a shushing gesture detection device, the device comprising:
[0028] An acquisition unit, configured to acquire facial feature points in a facial image including a human face in a vehicle interior captured by a vehicle-mounted camera;
[0029] A capture unit, configured to capture a first target image from the face image according to a rectangular area constructed with a target feature point among the face feature points as the center, a first preset multiple of the area of a circumscribed rectangle of a figure formed by the face feature points as the width, a preset ratio of the width as the height, and an extension direction of the face in the face image as the inclination direction of the height, wherein the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to a feature point of a lip area;
[0030] a determination unit, configured to input the first target image into a booing detection model to determine whether the face image contains a booing gesture;
[0031] The control unit is used to control a target controllable component in the vehicle to perform a preset action when it is determined that the face image contains the shushing gesture.
[0032] Optionally, the device further comprises:
[0033] A processing unit is used to input the face image into a face detection model after obtaining the face feature points to obtain initial face feature points in the face image; and to map the initial face feature points to the same feature points in a standard three-dimensional face model to obtain a reconstructed three-dimensional face model; and to map the reconstructed three-dimensional face model back to the same feature points in the face image to determine the target feature points.
[0034] Optionally, when the control unit is used to determine that the face image contains the shushing gesture, control the target controllable component in the vehicle to perform a preset action, including:
[0035] When it is determined that the face image contains the shushing gesture, a second target image is intercepted from the face image according to a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extending direction as the center, a second preset multiple of the face as the side length, and the extending direction as the inclined direction;
[0036] determining, according to the second target image, a shushing virtual slot in a region where the index finger is located when shushing is performed in the second target image;
[0037] Inputting the second target image into a positioning hand feature point model to obtain hand feature points;
[0038] determining whether the shushing gesture is a correct shushing gesture according to an index finger feature point among the hand feature points used to represent the index finger and a positional relationship with the shushing virtual slot;
[0039] If the shush gesture is correct, the target controllable component in the vehicle is controlled to perform a preset action.
[0040] Optionally, the target feature point is a lip feature point used to represent the center of the upper lip, and the three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the reconstructed three-dimensional model of the face as the origin, the direction of the line connecting the two eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane where the X-axis and the Z-axis are located as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system.
[0041] Optionally, the length of the virtual hushing groove is the distance from a nose tip feature point used to represent the nose tip among facial feature points to the lip feature point, the upper endpoint of the virtual hushing groove is located at 0.1 times the length of the nose tip feature along the positive direction of the Z axis, and the lower endpoint of the virtual hushing groove is located at 0.1 times the length of the lip feature point along the positive direction of the Z axis.
[0042] Optionally, the designated distance is 0.25 times the height of the face, and the second preset multiple is a value of 1.
[0043] Optionally, the first preset multiple is the square root of the area of the circumscribed rectangle, and the preset ratio is two-thirds.
[0044] Optionally, the control unit is used to control a target controllable component in a vehicle to perform a preset action, comprising:
[0045] When the target controllable component is a car phone, rejecting the current incoming call of the car phone;
[0046] When the target controllable component is an in-vehicle multimedia, pausing the content currently being played by the in-vehicle multimedia;
[0047] When the target controllable component is a vehicle window, the vehicle window is closed.
[0048] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the hushing gesture detection method described in any one of the first aspects above.
[0049] In a fourth aspect, an embodiment of the present application provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the hushing gesture detection method described in any one of the first aspects above.
[0050] The technical solution provided by the embodiments of the present application may have the following beneficial effects:
[0051] In the present application, after obtaining the facial feature points, in order to reduce interference during recognition and improve recognition efficiency, a first target image including a mouth area is captured from the facial image using a constructed rectangular area, and then the first target image is input into a shushing detection model to determine whether the facial image contains a shushing gesture. If a shushing finger is contained, the target controllable component in the vehicle is controlled to perform a preset action. The above method is accomplished through gestures, which can complete control more quickly than the prior art, and is therefore beneficial to improving interaction efficiency. Furthermore, since the gesture method is less affected by the environment than the voice control method, it is beneficial to improving anti-interference capability during interaction in a vehicle cabin environment.
[0052] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0054] Figure 1 A schematic diagram of a process for detecting a shushing gesture provided in an embodiment of the present application;
[0055] Figure 2 A schematic diagram of a flow chart of another method for detecting a shushing gesture provided in an embodiment of the present application;
[0056] Figure 3A schematic diagram of a flow chart of another method for detecting a shushing gesture provided in an embodiment of the present application;
[0057] Figure 4 A schematic diagram of the structure of a shushing gesture detection device provided in an embodiment of the present application;
[0058] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0060] Figure 1 A flow chart of a method for detecting a shushing gesture provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:
[0061] Step 101: Based on a facial image including a human face inside a vehicle captured by a vehicle-mounted camera, facial feature points in the facial image are obtained.
[0062] Step 102: capture a first target image from the facial image based on a rectangular area constructed with a target feature point among the facial feature points as the center, a first preset multiple of the area of a circumscribed rectangle of a figure formed by the facial feature points as the width, a preset ratio of the width as the height, and an extension direction of the face in the facial image as the inclination direction of the height, wherein the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to a feature point of the lip area.
[0063] Step 103: Input the first target image into a booing detection model to determine whether the face image contains a booing gesture.
[0064] Step 104: When it is determined that the facial image contains the shushing gesture, control a target controllable component in the vehicle to perform a preset action.
[0065] Specifically, a vehicle-mounted camera is arranged in the vehicle cabin, which obtains a facial video of the driver's face in real time, extracts frames from the facial video, and then obtains a facial image. It is also possible to use each frame in the facial video as a facial image. After obtaining the facial image, facial feature points in the facial image are obtained, such as: mouth feature points, eyebrow feature points, eye feature points, facial contour feature points, etc.
[0066] After acquiring the facial feature points, in order to improve the processing speed and reduce unnecessary interference, a first target image including the lip area can be intercepted from the facial image based on the feature points of the lip area. The interception is centered on the feature points of the lip area. This can include intercepting the lip area, and then taking a first preset multiple of the area of the circumscribed rectangle of the figure formed by the facial feature points as the width, a preset ratio of the above width as the height, and an extension direction of the face in the facial image as the height to construct a rectangular area (inclined rectangle), and intercepting the first target image from the facial image. This can ensure that the intercepted first target image includes not only the lip area, but also the area in front of the lip area. Since the shushing gesture needs to be judged by the lips, fingers, and the relationship between the two, after intercepting the first target image, it can be determined whether the current facial image contains a shushing gesture. When it is determined that the shushing gesture is contained, the corresponding target controllable component can be controlled to perform a preset action, such as closing the window, closing the sunroof, turning off the sound, etc.
[0067] The present application is completed through gestures, which can complete the control more quickly than the existing technology, thus helping to improve the interaction efficiency. Moreover, since the gesture method is less affected by the environment than the voice control method, it is helpful to improve the anti-interference ability when interacting in the cabin environment.
[0068] In one possible embodiment, Figure 2 A flow chart of another method for detecting a shushing gesture provided in an embodiment of the present application is shown as follows: Figure 2 As shown, after obtaining the facial feature points, the method further includes the following steps:
[0069] Step 201: input the face image into a face detection model to obtain initial face feature points in the face image.
[0070] Step 202: Map the initial facial feature points to the same feature points in a standard three-dimensional face model to obtain a reconstructed three-dimensional face model.
[0071] Step 203: Map the reconstructed 3D face model back to the same feature point in the face image to determine the target feature point.
[0072] Specifically, since the three-dimensional feature points can provide more information about the position and orientation of the head in space, the system can work effectively even when the head has different rotation angles relative to the camera. For example, when the side face is required to make a shushing gesture, the index finger is 1 cm away from the lips, which is easier to determine using three-dimensional facial feature points. Since the on-board camera (a monocular camera) cannot directly capture the depth information in three-dimensional space, the solvePnP algorithm is used to estimate the three-dimensional coordinates corresponding to the facial feature points in the face image. This process starts with using a pre-trained deep learning model to detect the two-dimensional facial feature points in the input image. These two-dimensional feature points usually include significant facial feature points such as the corners of the eyes, the tip of the nose, and the corners of the mouth. Their positions in the image can be accurately detected by the deep learning model. In a two-dimensional image, the origin of the coordinate system can be the upper left corner of the image, where the x-axis extends to the right and the y-axis extends downward. The coordinate unit is pixel, and the solvePnP algorithm is used for head posture estimation. In order to map these two-dimensional feature points to three-dimensional space, a standard three-dimensional face reference model is required, namely: the standard three-dimensional face model. The origin of the three-dimensional coordinate system corresponding to the standard three-dimensional face model is the tip of the nose, the left side of the line connecting the two eyes is the x-axis, the front side is the z-axis, and the vertical direction of the x-axis and z-axis is the y-axis (this direction is the extension direction of the face in the face image). This reference model contains the three-dimensional coordinate points corresponding to the two-dimensional feature points. Then, we use the solvePnP algorithm to estimate the posture of the face relative to the camera. It can calculate the best rotation and translation vectors based on the feature points on the two-dimensional image and the corresponding points in the three-dimensional model, thereby converting the two-dimensional feature points into the three-dimensional coordinate system. After calculating the three-dimensional coordinates, the accuracy of the reconstruction can be verified by projecting these coordinates back to the two-dimensional plane. The accuracy of the three-dimensional coordinates can be further improved by iterative optimization algorithms, such as the Levenberg-Marquardt method. If the projected two-dimensional points are closely aligned with the feature points detected in the original image, then the three-dimensional reconstruction can be considered successful. In this way, an accurate three-dimensional face model can be reconstructed from an ordinary two-dimensional image.
[0073] In one possible embodiment, Figure 3 A flow chart of another method for detecting a shushing gesture provided in an embodiment of the present application is shown as follows: Figure 3 As shown, when executing step 104, it can be achieved by the following steps:
[0074] Step 301: When it is determined that the face image contains the shushing gesture, a second target image is captured from the face image according to a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extension direction as the center, a second preset multiple of the face as the side length, and the extension direction as the inclined direction.
[0075] Step 302: Determine, according to the second target image, a shushing virtual slot in the area where the index finger is located when shushing in the second target image.
[0076] Step 303: input the second target image into the hand feature point positioning model to obtain hand feature points.
[0077] Step 304: Determine whether the shushing gesture is a correct shushing gesture based on the index finger feature point among the hand feature points used to represent the index finger and its positional relationship with the shushing virtual slot.
[0078] Step 305: If it is a correct shushing gesture, control the target controllable component in the vehicle to perform a preset action.
[0079] Specifically, in order to prevent the shushing gesture determined in step 103 from being wrong, for example, recognizing a straw, a sausage, a cigarette, etc. as an index finger, and thus misjudging it as a shushing gesture, and whether the gesture conforms to the shushing gesture, that is, whether the index finger feature point of the index finger and the position relationship with the shushing virtual slot conform to the shushing gesture requirements, the following points are specifically included:
[0080] 1. Whether the index finger is straightened. The index finger includes four key points: fingertip, first knuckle, second knuckle and base of third knuckle. According to the four key points of the index finger, three bone segment vectors are calculated, namely the vectors from the fingertip to the first knuckle, from the first knuckle to the second knuckle, and from the second knuckle to the base of the third knuckle. The angle between adjacent bone segment vectors is calculated, and a threshold value (such as 10 degrees) is set. If all angles are less than the threshold value, the index finger is judged to be straightened.
[0081] 2. Is the index finger parallel to the vertical midline of the face and within a certain distance in front of the face? Project the virtual slot of the hissing sound onto the imaging plane to form a convex hull with the four feature points of the index finger. When the area of the convex hull is less than 1 / 40 of the area of the face, the index finger is parallel to the vertical midline of the face.
[0082] 3. Is the index finger pointing upward? Let the vector from the base of the third knuckle of the index finger to the tip of the index finger be A, and the projection vector from the lower end to the upper end of the virtual slot on the image plane be B. When the angle between A and B is less than 90 degrees, the index finger is pointing upward.
[0083] 4. Whether the index finger is located at the horizontal center of the face. The geometric center of the four key points of the index finger is between the areas formed by the two third points of the lips along the numerical direction of the face.
[0084] The relative positions of the shushing virtual groove and the index finger are determined by projecting the shushing virtual groove onto the screen plane and determining the angle between the projection of the shushing virtual groove and the two line segments of the index finger and the end point distance.
[0085] In order to quickly complete the above detection, it is necessary to intercept the lip area and the area where the index finger may exist in the face image, that is, take the three-dimensional feature point at the center of the upper lip, move it 0.25 head height along the negative direction of the three-dimensional y axis, and project it on the image plane to determine the center point of the diamond area. The side length of the diamond area is equal to the height of a head, and the height direction (vertical direction) of the diamond area is parallel to the vertical projection of the three-dimensional face feature point on the image plane. The hand feature point refers to the specific key position in the hand image, including the palm and the joints and fingertips of each finger. The output of the deep learning model includes two parts: binary classification, which refers to whether there is a hand in the picture. The coordinates of the key points of the hand. If the hand is detected, the coordinate information of each key point of the hand will be output, including the palm and the joints and fingertips of all fingers.
[0086] Based on the 3D face model, the ideal position of the index finger to trigger the shushing sound is calculated. This position is called the "shushing sound virtual slot". The input is the coordinates of the 3D feature points of the face, and the output is the coordinates of the shushing sound virtual slot. The shushing sound virtual slot is a 3D line segment represented by two 3D points, and its coordinates are calculated using the 3D feature points of the face. The length of the shushing sound virtual slot is the distance from the tip of the nose to the midpoint of the upper lip. The upper end point of the shushing sound virtual slot is located at the tip of the nose and is moved 1 / 10 of the length of the shushing sound virtual slot along the positive direction of the z-axis. The lower end point is located at the midpoint of the upper lip and is moved 1 / 10 of the length of the shushing sound virtual slot along the positive direction of the z-axis.
[0087] After obtaining the shushing virtual groove and the feature point of the index finger, whether the shushing gesture is a correct shushing gesture is determined based on the feature point of the index finger and its positional relationship with the shushing virtual groove.
[0088] In a feasible implementation scheme, the target feature point is a lip feature point used to represent the center of the upper lip, and the three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the three-dimensional face reconstructed model as the origin, the direction of the line connecting the two eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane where the X-axis and the Z-axis are located as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system.
[0089] In a feasible implementation manner, the length of the virtual hushing groove is the distance from the nose tip feature point used to represent the nose tip among the facial feature points to the lip feature point (specifically, the center of the lip), the upper endpoint of the virtual hushing groove is located at 0.1 times the length of the virtual hushing groove of the nose tip feature along the positive direction of the Z axis, and the lower endpoint of the virtual hushing groove is located at 0.1 times the length of the virtual hushing groove of the lip feature point along the positive direction of the Z axis.
[0090] In a feasible implementation manner, the designated distance is 0.25 times the height of the face, and the second preset multiple is a value of 1.
[0091] In a feasible implementation manner, the first preset multiple is the square root of the area of the circumscribed rectangle, and the preset ratio is two-thirds.
[0092] In a feasible implementation manner, the control of the target controllable component in the vehicle to perform a preset action includes:
[0093] When the target controllable component is a car phone, rejecting the current incoming call of the car phone;
[0094] When the target controllable component is an in-vehicle multimedia, pausing the content currently being played by the in-vehicle multimedia;
[0095] When the target controllable component is a vehicle window, the vehicle window is closed.
[0096] It should be noted that the preset action executed can also shut down or temporarily set other controllable components, such as the vehicle air conditioner. The specific controllable components and preset actions can be set according to actual needs and are not specifically limited here.
[0097] Figure 4 A schematic diagram of the structure of a shushing gesture detection device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device comprises:
[0098] An acquisition unit 41 is used to acquire facial feature points in a facial image including a human face in a vehicle interior captured by a vehicle-mounted camera;
[0099] The interception unit 42 is used to intercept a first target image from the face image according to a rectangular area constructed with a target feature point among the face feature points as the center, a first preset multiple of the area of a circumscribed rectangle of a figure formed by the face feature points as the width, a preset ratio of the width as the height, and an extension direction of the face in the face image as the inclined direction of the height, wherein the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to a feature point of a lip area;
[0100] A judging unit 43, configured to input the first target image into a booing detection model to judge whether the face image contains a booing gesture;
[0101] The control unit 44 is configured to control a target controllable component in the vehicle to perform a preset action when it is determined that the face image contains the shushing gesture.
[0102] In a feasible embodiment, the device further comprises:
[0103] A processing unit is used to input the face image into a face detection model after obtaining the face feature points to obtain initial face feature points in the face image; and to map the initial face feature points to the same feature points in a standard three-dimensional face model to obtain a reconstructed three-dimensional face model; and to map the reconstructed three-dimensional face model back to the same feature points in the face image to determine the target feature points.
[0104] In a feasible implementation, the control unit is used to control the target controllable component in the vehicle to perform a preset action when it is determined that the face image contains the shushing gesture, including:
[0105] When it is determined that the face image contains the shushing gesture, a second target image is intercepted from the face image according to a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extending direction as the center, a second preset multiple of the face as the side length, and the extending direction as the inclined direction;
[0106] determining, according to the second target image, a shushing virtual slot in a region where the index finger is located when shushing is performed in the second target image;
[0107] Inputting the second target image into a positioning hand feature point model to obtain hand feature points;
[0108] determining whether the shushing gesture is a correct shushing gesture according to an index finger feature point among the hand feature points used to represent the index finger and a positional relationship with the shushing virtual slot;
[0109] If the shush gesture is correct, the target controllable component in the vehicle is controlled to perform a preset action.
[0110] In a feasible implementation scheme, the target feature point is a lip feature point used to represent the center of the upper lip, and the three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the three-dimensional face reconstructed model as the origin, the direction of the line connecting the two eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane where the X-axis and the Z-axis are located as the Y-axis, and the extension direction is the Y-axis direction of the three-dimensional coordinate system.
[0111] In a feasible implementation manner, the length of the virtual hushing groove is the distance from the nose tip feature point used to represent the nose tip among the facial feature points to the lip feature point, the upper endpoint of the virtual hushing groove is located at 0.1 times the length of the nose tip feature along the positive direction of the Z axis, and the lower endpoint of the virtual hushing groove is located at 0.1 times the length of the lip feature point along the positive direction of the Z axis.
[0112] In a feasible implementation manner, the designated distance is 0.25 times the height of the face, and the second preset multiple is a value of 1.
[0113] In a feasible implementation manner, the first preset multiple is the square root of the area of the circumscribed rectangle, and the preset ratio is two-thirds.
[0114] In a feasible implementation, the control unit is used to control a target controllable component in a vehicle to perform a preset action, including:
[0115] When the target controllable component is a car phone, rejecting the current incoming call of the car phone;
[0116] When the target controllable component is an in-vehicle multimedia, pausing the content currently being played by the in-vehicle multimedia;
[0117] When the target controllable component is a vehicle window, the vehicle window is closed.
[0118] For the principle description of the shushing gesture detection device, please refer to the relevant explanation of the shushing gesture detection method, which will not be described in detail here.
[0119] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application includes: a processor 501, a storage medium 502 and a bus 503, wherein the storage medium 502 stores machine-readable instructions executable by the processor 501. When the electronic device runs a method for detecting a hushing gesture as in the embodiment, the processor 501 communicates with the storage medium 502 through the bus 503, and the processor 501 executes the machine-readable instructions to perform the steps in the embodiment.
[0120] In an embodiment, the storage medium 502 may also execute other machine-readable instructions to execute other methods described in the embodiment. For specific execution method steps and principles, please refer to the description of the embodiment, which will not be described in detail here.
[0121] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps shown in the above embodiment are executed.
[0122] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0124] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0125] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.
[0126] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.
[0127] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for detecting a shushing gesture, characterized in that: The method comprises: According to a facial image including a human face inside a vehicle captured by an onboard camera, acquiring facial feature points in the facial image; A first target image is intercepted from the face image according to a rectangular area constructed with a target feature point among the face feature points as the center, a first preset multiple of the area of a circumscribed rectangle of a figure formed by the face feature points as the width, a preset ratio of the width as the height, and an extension direction of the face in the face image as the inclination direction of the height, wherein the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to a feature point of the lip area; Inputting the first target image into a shushing detection model to determine whether the face image contains a shushing gesture; When it is determined that the face image contains the shushing gesture, controlling a target controllable component in the vehicle to perform a preset action; Wherein, when it is determined that the face image contains the shushing gesture, controlling the target controllable component in the vehicle to perform a preset action includes: When it is determined that the face image contains the shushing gesture, a second target image is intercepted from the face image according to a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extending direction as the center, a second preset multiple of the face as the side length, and the extending direction as the inclined direction; determining, according to the second target image, a shushing virtual slot in a region where the index finger is located when shushing is performed in the second target image; Inputting the second target image into a positioning hand feature point model to obtain hand feature points; determining whether the shushing gesture is a correct shushing gesture according to an index finger feature point among the hand feature points used to represent the index finger and a positional relationship with the shushing virtual slot; If the shush gesture is correct, the target controllable component in the vehicle is controlled to perform a preset action.
2. The method according to claim 1, characterized in that After obtaining the facial feature points, the method further includes: Inputting the face image into a face detection model to obtain initial face feature points in the face image; Mapping the initial facial feature points to the same feature points in a standard three-dimensional face model to obtain a reconstructed three-dimensional face model; The reconstructed three-dimensional face model is mapped back to the same feature point in the face image to determine the target feature point.
3. The method according to claim 1, characterized in that The target feature point is a lip feature point used to represent the center of the upper lip. The three-dimensional coordinate system is constructed with the nose tip feature point representing the nose tip in the reconstructed three-dimensional model of the face as the origin, the direction of the line connecting the two eyes as the X-axis, the facial orientation as the Z-axis, and the direction perpendicular to the plane where the X-axis and the Z-axis are located as the Y-axis. The extension direction is the Y-axis direction of the three-dimensional coordinate system.
4. The method according to claim 3, characterized in that The length of the virtual hushing groove is the distance from the nose tip feature point used to represent the nose tip among the facial feature points to the lip feature point, the upper endpoint of the virtual hushing groove is located at 0.1 times the length of the nose tip feature along the positive direction of the Z axis, and the lower endpoint of the virtual hushing groove is located at 0.1 times the length of the lip feature point along the positive direction of the Z axis.
5. The method according to claim 1, characterized in that The specified distance is 0.25 times the height of the face, and the second preset multiple is a value of 1.
6. The method according to claim 1, characterized in that The first preset multiple is the square root of the area of the circumscribed rectangle, and the preset ratio is two-thirds.
7. The method according to claim 1, characterized in that The controlling of the target controllable component in the vehicle to perform a preset action includes: When the target controllable component is a car phone, rejecting the current incoming call of the car phone; When the target controllable component is an in-vehicle multimedia, pausing the content currently being played by the in-vehicle multimedia; When the target controllable component is a vehicle window, the vehicle window is closed.
8. A shushing gesture detection device, characterized in that: The device comprises: An acquisition unit, configured to acquire facial feature points in a facial image including a human face in a vehicle interior captured by a vehicle-mounted camera; A capture unit, configured to capture a first target image from the face image according to a rectangular area constructed with a target feature point among the face feature points as the center, a first preset multiple of the area of a circumscribed rectangle of a figure formed by the face feature points as the width, a preset ratio of the width as the height, and an extension direction of the face in the face image as the inclination direction of the height, wherein the first preset multiple is a positive number, the preset ratio is less than 1, and the target feature point belongs to a feature point of a lip area; a determination unit, configured to input the first target image into a booing detection model to determine whether the face image contains a booing gesture; a control unit, configured to control a target controllable component in the vehicle to perform a preset action when it is determined that the face image contains the shushing gesture; The control unit is used to control the target controllable component in the vehicle to perform a preset action when it is determined that the face image contains the shushing gesture, including: When it is determined that the face image contains the shushing gesture, a second target image is intercepted from the face image according to a rhombus constructed with a specified distance from the target feature point in the opposite direction of the extending direction as the center, a second preset multiple of the face as the side length, and the extending direction as the inclined direction; determining, according to the second target image, a shushing virtual slot in a region where the index finger is located when shushing is performed in the second target image; Inputting the second target image into a positioning hand feature point model to obtain hand feature points; determining whether the shushing gesture is a correct shushing gesture according to an index finger feature point among the hand feature points used to represent the index finger and a positional relationship with the shushing virtual slot; If the shush gesture is correct, the target controllable component in the vehicle is controlled to perform a preset action.
9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method for detecting a shushing gesture according to any one of claims 1 to 7.
10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the shushing gesture detection method according to any one of claims 1 to 7.
Citation Information
Cited By
Shushing gesture detection method and apparatus, electronic device, and storage medium
EP4730281A1
