Palm position determination method, device, electronic device and storage medium

Abstract

The present application provides a palm position determination method, device, electronic device and storage medium. The method first detects the key points of the target palm, performs linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger, obtains linear fitting lines corresponding to multiple fingers, determines the inclination of the target palm by multiple linear fitting lines, and obtains a position frame covering all key points of the target palm in combination with the positions of the key points, which can indicate the size and position of the target palm. The inclination of the palm is analyzed based on the key points obtained by the preliminary detection, and the position frame is obtained based on the inclination and combined with all the key points obtained, which can ensure that the inclination and size of the generated position frame are basically consistent with the target palm, and the positioning is more accurate. When gesture recognition and continuous tracking of the target palm are performed based on the position frame, the interference of the background environment in the image or the omission of key points can be reduced, ensuring that the feature extraction is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a palm position determination method, device, electronic equipment and storage medium. Background Art

[0002] Due to the development of computer vision technology, gesture control based on computer vision has become a widely used interaction method in the field of human-computer interaction. Gesture control usually uses a camera to track human motion and behavior, and sends control instructions to the device through human motion posture. In this way, it is possible to remotely control the device with bare hands within a certain distance without the need for control devices such as remote controls, or traditional input and output devices such as mouse and keyboard. However, many gesture recognition scenarios are very dependent on accurately framing the position and size of the hand, and the currently commonly used deep learning-based target detection algorithm can usually only roughly frame the position and size of the hand, and its accuracy is not high, which will also affect the accuracy of subsequent gesture recognition. Therefore, how to accurately locate the palm position is a problem worth studying. Summary of the invention

[0003] Based on the above situation, the main purpose of this application is to provide a palm position determination method, device, electronic device and storage medium, which can analyze the inclination of the hand according to the key points of the palm, and determine a rectangular frame that can locate the palm position based on the inclination to determine the size and position of the palm.

[0004] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0005] A method for determining a palm position, the method comprising:

[0006] Performing key point detection on the image to be detected containing the target palm to determine each key point of the target palm;

[0007] Performing linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger to obtain linear fitting lines corresponding to the multiple fingers;

[0008] Determining the inclination of the target palm according to the slopes of the plurality of linear fitting lines;

[0009] According to the inclination and the positions of the key points of the target palm, a position frame indicating the size and position of the target palm is obtained, and the position frame covers all the key points of the target palm.

[0010] In a possible implementation, before performing key point detection on the image to be detected including the target palm, the method further includes:

[0011] Pre-position the target palm of the acquired image through the target detection and positioning model to obtain a pre-positioning result;

[0012] A sub-image including the target palm is cropped from the acquired image according to the pre-positioning result to obtain the image to be detected.

[0013] In a possible implementation manner, determining the inclination of the target palm according to the slopes of the plurality of linear fitting lines includes:

[0014] Determining a plurality of corresponding inclination angles according to the slopes of the plurality of linear fitting straight lines;

[0015] After removing outliers from the plurality of tilt angles, a plurality of valid tilt angles are obtained;

[0016] An average value is calculated for the multiple effective inclination angles to obtain an average angle, and the average angle is determined to be the inclination of the target palm.

[0017] In a possible implementation, obtaining a position frame indicating the size and position of the target palm according to the inclination and the positions of the key points of the target palm includes:

[0018] Determining the direction of the target palm according to the inclination;

[0019] If the direction of the target palm is vertical or horizontal, calculate the positions of four vertices of the initial detection frame according to the positions of the key points of the target palm, and determine the position frame according to the initial detection frame;

[0020] If the direction of the target palm is not vertical or horizontal, the functions corresponding to the four sides of the initial detection frame are calculated, and the position frame is determined according to the initial detection frame.

[0021] In a possible implementation manner, determining the direction of the target palm according to the inclination includes:

[0022] When the inclination meets the first preset condition, determining that the direction of the target palm is vertical or horizontal;

[0023] When the inclination does not meet the first preset condition, it is determined that the direction of the target palm is not vertical or horizontal, wherein the first preset condition is: the absolute value of the slope value corresponding to the inclination is greater than or equal to the first preset threshold T1 or less than 1 / T1.

[0024] In a possible implementation, calculating the positions of four vertices of the initial detection frame according to the positions of the key points of the target palm, and determining the position frame according to the initial detection frame includes:

[0025] The minimum horizontal coordinate value among all key points is taken as the first horizontal coordinate value Xs, and the maximum horizontal coordinate value is taken as the second horizontal coordinate value Xg;

[0026] The minimum ordinate value among all key points is taken as the first ordinate value Ys, and the maximum ordinate value is taken as the second ordinate value Yg;

[0027] Taking (Xs, Ys), (Xs, Yg), (Xg, Ys), (Xg, Yg) as the coordinates of each vertex of the initial detection frame;

[0028] After the initial detection frame is enlarged, the position frame is obtained.

[0029] In a possible implementation, calculating the functions corresponding to the four sides of the initial detection frame and determining the position frame according to the initial detection frame includes:

[0030] Calculate the first maximum intercept Pg and the first minimum intercept Ps determined when a straight line with an average slope K passes through all key points of the target palm; the average slope K is determined according to the inclination of the target palm;

[0031] Calculate the second maximum intercept Tg and the second minimum intercept Ts of a straight line with the negative reciprocal of the average slope K as the slope when the straight line passes through all key points of the target palm;

[0032] Determine four functions corresponding to the four sides of the initial detection frame according to the first maximum intercept Pg, the first minimum intercept Ps, the second maximum intercept Tg, the second minimum intercept Ts and the average slope K, the four functions being: y=x*K+Ps, y=x*K+Pg, y=-x / K+Ts, y=-x / K+Tg;

[0033] Using the rectangular frame surrounded by the four functions as the initial detection frame;

[0034] After the initial detection frame is enlarged, the position frame is obtained.

[0035] In a possible implementation manner, obtaining a position frame indicating the size and position of the target palm includes:

[0036] Generate a rectangular box covering all key points of the target palm;

[0037] The position frame is obtained by enlarging the rectangular frame.

[0038] To achieve the above object, the present application further provides a palm position determination device, the palm position determination device comprising:

[0039] A key point detection module is used to perform key point detection on the image to be detected containing the target palm, and determine each key point of the target palm;

[0040] A linear fitting module, used for performing linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger, to obtain linear fitting lines corresponding to multiple fingers;

[0041] An inclination determination module, used to determine the inclination of the target palm according to the slopes of the plurality of linear fitting lines;

[0042] The position frame generating module is used to obtain a position frame indicating the size and position of the target palm according to the inclination and the positions of the key points of the target palm, wherein the position frame covers all the key points of the target palm.

[0043] In a possible implementation, the palm position determining device may further include:

[0044] An image pre-positioning module is used to pre-position the target palm of the acquired image through a target detection and positioning model to obtain a pre-positioning result;

[0045] A sub-image acquisition module is used to crop a sub-image containing the target palm from the acquired image according to the pre-positioning result to obtain the image to be detected.

[0046] In a possible implementation, the position frame generating module generates a position frame covering all key points of the target palm, including:

[0047] Generate a rectangular frame covering all key points of the target palm;

[0048] The position frame is obtained by enlarging the rectangular frame.

[0049] In a possible implementation, the inclination determination module includes:

[0050] An inclination angle determination unit, configured to determine a plurality of corresponding inclination angles according to the slopes of the plurality of linear fitting lines;

[0051] A validity judgment unit, configured to obtain a plurality of valid tilt angles after removing outliers from the plurality of tilt angles;

[0052] The inclination calculation unit is used to calculate an average value of the multiple effective inclination angles to obtain an average angle, and determine the average angle as the inclination of the target palm.

[0053] In a possible implementation, the position box generating module includes:

[0054] a direction determining unit, configured to determine the direction of the target palm according to the inclination;

[0055] a position frame calculation unit, configured to calculate the positions of four vertices of an initial detection frame according to the positions of key points of the target palm when the direction of the target palm is vertical or horizontal, and determine the position frame according to the initial detection frame; and

[0056] When the direction of the target palm is not vertical or horizontal, the functions corresponding to the four sides of the initial detection frame are calculated, and the position frame is determined according to the initial detection frame.

[0057] In a possible implementation, the direction determination unit performing direction determination includes:

[0058] When the inclination meets the first preset condition, determining that the direction of the target palm is vertical or horizontal;

[0059] When the inclination does not meet the first preset condition, it is determined that the direction of the target palm is not vertical or horizontal, wherein the first preset condition is: the absolute value of the slope value corresponding to the inclination is greater than or equal to the first preset threshold T1 or less than 1 / T1.

[0060] In a possible implementation, the position frame calculation unit includes:

[0061] The coordinate calculation subunit is used for: when the direction of the target palm is vertical or horizontal:

[0062] The minimum horizontal coordinate value among all key points is taken as the first horizontal coordinate value Xs, and the maximum horizontal coordinate value is taken as the second horizontal coordinate value Xg;

[0063] The minimum ordinate value among all key points is taken as the first ordinate value Ys, and the maximum ordinate value is taken as the second ordinate value Yg;

[0064] Taking (Xs, Ys), (Xs, Yg), (Xg, Ys), (Xg, Yg) as the coordinates of each vertex of the initial detection frame;

[0065] Amplification subunit, used for:

[0066] After the initial detection frame is enlarged, the position frame is obtained.

[0067] In a possible implementation manner, the position frame calculation unit further includes:

[0068] A function obtaining unit is used for, when the direction of the target palm is not vertical or horizontal:

[0069] Calculate the first maximum intercept Pg and the first minimum intercept Ps determined when a straight line with an average slope K passes through all key points of the target palm; the average slope K is determined according to the inclination of the target palm;

[0070] Calculate the second maximum intercept Tg and the second minimum intercept Ts of a straight line with the negative reciprocal of the average slope K as the slope when the straight line passes through all key points of the target palm;

[0071] Determine four functions corresponding to the four sides of the initial detection frame according to the first maximum intercept Pg, the first minimum intercept Ps, the second maximum intercept Tg, the second minimum intercept Ts and the average slope K, the four functions being: y=x*K+Ps, y=x*K+Pg, y=-x / K+Ts, y=-x / K+Tg;

[0072] The rectangular frame enclosed by the four functions is used as the initial detection frame.

[0073] To achieve the above objectives, the present application also provides an electronic device, including a processor, wherein the processor is used to implement the palm position determination method as described above.

[0074] To achieve the above objectives, the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the palm position determination method as described above is executed.

[0075] Beneficial effects:

[0076] The embodiment of the present application adopts the above-mentioned palm position determination method, first detects the key points of the target palm, performs linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger, obtains linear fitting lines corresponding to multiple fingers, determines the inclination of the target palm by multiple linear fitting lines, and obtains a position frame covering all key points of the target palm in combination with the positions of each key point, which can indicate the size and position of the target palm. The inclination of the palm is analyzed based on the key points obtained by the preliminary detection, and the position frame is defined based on the inclination and the key points obtained, which can ensure that the inclination and size of the obtained position frame are basically consistent with the target palm, and the positioning is more accurate. When gesture recognition and continuous tracking of the target palm are performed based on the position frame, the interference of the background environment in the image can be reduced or key points can be avoided from being missed, ensuring that the feature extraction is more accurate.

[0077] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The preferred embodiments according to the present invention will be described below with reference to the accompanying drawings.

[0079] Figure 1 The figure is a schematic diagram of the flow of a palm position determination method in one embodiment of the present application;

[0080] Figure 2 The figure shows a schematic diagram of 21 key points of a palm in one embodiment of the present application;

[0081] Figure 3 The figure shows a schematic diagram of linear fitting of key points of each finger of the target palm in one embodiment of the present application;

[0082] Figure 4 The following is an example of an embodiment of the present application. Figure 3 Schematic diagram of the position frame generated by the target palm;

[0083] Figure 5 The figure is a functional block diagram of a palm position determination device in one embodiment of the present application;

[0084] Figure 6 The figure shows a schematic diagram of the structure of an electronic device used to implement the embodiment disclosed in the present application. DETAILED DESCRIPTION

[0085] In order to explain the technical solution of the present application in more detail and to facilitate further understanding of the present application, the specific implementation methods of the present application are described below in conjunction with the accompanying drawings. However, it should be understood that all illustrative embodiments and their descriptions are used to explain the present application and do not constitute the sole limitation of the present application.

[0086] In the present application, “first”, “second”, “third”, “fourth”, etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance.

[0087] To facilitate understanding of the various embodiments disclosed in the present application, the palm position determination method disclosed in the various embodiments of the present application is first introduced in detail. The execution body of the palm position determination method provided in the various embodiments of the present application can generally be executed by a computing device, which can be implemented as software, or as a combination of software and hardware. The computing device can be integrated in an electronic device with a certain computing capability, and the electronic device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a mobile phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a mobile terminal such as a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal), and a fixed terminal such as a digital TV, a desktop computer, an intelligent interactive display device, etc. In some possible implementations, the palm position determination method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0088] Please refer to Figure 1 FIG. 1 is a flow chart of a palm position determination method provided by an embodiment of the present application, wherein the method comprises steps S100-S400:

[0089] S100, performing key point detection on the image to be detected including the target palm, and determining each key point of the target palm;

[0090] S200, performing linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger to obtain linear fitting lines corresponding to the multiple fingers;

[0091] S300, determining the inclination of the target palm according to the slopes of the plurality of linear fitting lines;

[0092] S400, obtaining a position frame indicating the size and position of the target palm according to the inclination and the positions of the key points of the target palm, wherein the position frame covers all the key points of the target palm.

[0093] The embodiment of the present application adopts the above method, first detects the key points of the target palm, performs linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger, obtains linear fitting lines corresponding to multiple fingers, determines the inclination of the target palm by multiple linear fitting lines, and generates a position frame covering all key points of the target palm in combination with the positions of each key point. The inclination of the palm is analyzed based on the key points obtained by preliminary detection, and the position frame is defined based on the inclination and in combination with all the key points obtained, which can ensure that the inclination and size of the obtained position frame are basically consistent with the target palm, and the positioning is more accurate. When performing gesture recognition and continuous tracking of the target palm based on the position frame, the interference of the background environment in the background image can be reduced or key points can be avoided from being missed, ensuring that the feature extraction is more accurate.

[0094] The following is a detailed description of steps S100-S400:

[0095] In step S100, key point detection can be performed using a pre-trained key point detection model. The key point detection model can use a deep neural network model. Exemplarily, a deep neural network model such as a ResNet network structure, a MobileNet network structure, or a NASNet network structure can be used.

[0096] The key point detection model can be a key point detection model with different numbers. For example, it can be a 21 key point detection model, such as Figure 2 As shown, it is a schematic diagram of 21 common key points of the palm. In the subsequent embodiments of this application, the key point detection model outputs 21 key points as an example for explanation, but it does not mean that only 21 key points can be used in each embodiment of this application. The palm position determination method of this application can be applied to the key point detection model using any number of key points, for example, it can also be 14 key points.

[0097] In this embodiment, before step S100, the following steps may also be included:

[0098] S101, pre-positioning the target palm on the acquired image using a target detection and positioning model to obtain a pre-positioning result;

[0099] S102: cropping a sub-image including the target palm from the acquired image according to the pre-positioning result to obtain the image to be detected.

[0100] In the above steps S101 and S102, a camera is usually used to shoot a dynamic video of the actions of the controller of the electronic device. Images can be obtained frame by frame from the dynamic video, and each frame of the image is preprocessed, and the preprocessing includes operations such as scaling and enhancement. The target detection and positioning model can be a pre-trained neural network model. For example, neural network models such as Yolo-v1 / 2 / 3 / 4 network structure, SSD network structure, FPN network structure, etc. can be used. The trained target detection and positioning model can pre-position the target palm in the acquired image. According to the result of the pre-positioning, a relatively rough detection frame can usually be generated, and the position of the target palm can be included in the frame, but the detection frame is usually relatively large and contains more background information. A sub-image is directly cut out from the image to be positioned in the acquired image according to the detection frame containing the target palm. The sub-image can be used as the input image of the subsequent key point detection model, that is, the image to be detected.

[0101] It can be understood that in the present invention, "generation" does not mean that the detection box or position box needs to be actually displayed, including but not limited to obtaining corresponding data that can determine the specific position and range of the detection box or position box, or directly drawing the detection box or position box and displaying it.

[0102] In the above steps S101 and S102, the image to be located is first pre-located through the trained neural network model and then the image to be detected containing the target palm is cropped. This can reduce the amount of image data that needs to be processed during subsequent key point detection, speed up the detection of key points, and avoid the contamination of more irrelevant data affecting the detection results.

[0103] In step S200, the key point at the base of the target palm is taken and linearly fitted with the four key points on each of the other fingers to obtain a linear fitting line corresponding to each finger. In this embodiment, under the premise that all five key points are used, the linear fitting method preferably adopts the least square method for fitting. It is understandable that in other embodiments, other linear fitting methods may also be used, including but not limited to the gradient descent method, the gradient descent method, the Gauss-Newton method, etc.

[0104] In this embodiment, a corresponding linear fitting line may be obtained for each of the five fingers, or a corresponding linear fitting line may be obtained for only some of the fingers.

[0105] In step S300, the slope of each linear fitting line corresponds to an inclination angle, which represents the angle between the finger and the horizontal axis x of the plane coordinate system where the target palm is located. For multiple inclination angles, an average angle can be obtained by averaging. In this application, the angle value or the slope value can be used as a parameter to measure the inclination of the target palm. Therefore, the average angle, or the average slope corresponding to the average angle, can be used as the inclination of the target palm.

[0106] In step S400, after the inclination of the target palm is determined, a rectangular frame with the same inclination can be drawn based on the inclination. At the same time, according to the position of each key point, the size of the target palm can also be analyzed and determined, thereby determining the minimum area and center position of the rectangular frame, so that all key points are contained in the rectangular frame. Exemplarily, a center point can be determined by the top, bottom, rightmost and leftmost points of all key points, and the center point is used as the center point of the rectangular frame, and the four sides of the rectangular frame at least include the above four points to ensure that the rectangular frame covers all key points. The above rectangular frame can be used directly as a position frame, or it can be enlarged by a preset multiple as a position frame. The accuracy of the position frame is higher. In subsequent gesture recognition or continuous tracking of the target palm, the recognition result of the position frame can be used as the basis, and a smaller image can be further cropped from the image to be detected and input into other subsequent models. In this way, the amount of calculation can be reduced, the interference of the background environment in the background image can be reduced, or key points can be avoided from being missed, and the feature extraction can be ensured to be more accurate.

[0107] In an optional embodiment, in step S200, considering that the thumb is generally more flexible and is usually not in the same direction as the other four fingers, and is usually not in the same direction as the palm, it is not helpful for confirming the direction of the palm. Therefore, only the corresponding linear fitting lines can be obtained for the other four thumbs except the thumb. For example, Figure 3 As shown, it is a schematic diagram of linear fitting of the key points of each finger of the target palm in this embodiment, such as Figure 3 As shown, an xy coordinate system can be established in the image to be detected, and linear fitting can be performed on the four key points on each finger and the key points at the base of the palm to obtain four linear fitting lines L1, L2, L3, and L4.

[0108] By adopting the solution of the above embodiment, the number of linear fitting calculations can be reduced, and a large error in determining the inclination of the target palm can be avoided, so that the position frame finally determined is more accurate.

[0109] In an optional embodiment, step S300 includes the following steps:

[0110] S301, determining a plurality of corresponding inclination angles according to the slopes of the plurality of linear fitting lines;

[0111] S302, removing outliers from the plurality of tilt angles to obtain a plurality of valid tilt angles;

[0112] S303, calculating an average value of the multiple valid inclination angles to obtain an average angle, and determining the average angle as the inclination of the target palm.

[0113] In the above steps S301-S303, the slope of each linear fitting line can determine the inclination angle of the line. Figure 3 Take the four linear fitting lines L1, L2, L3, and L4 in as an example, which correspond to four tilt angles A1, A2, A3, and A4 respectively. For A1, A2, A3, and A4, the outliers are analyzed by clustering algorithm, the outliers are removed, and the remaining ones are determined as valid tilt angles. Exemplarily, the average value of A1, A2, A3, and A4 can be first calculated, and the Manhattan distance L1 from A1, A2, A3, and A4 to the average value can be calculated respectively to obtain the average Manhattan distance Lm. If for a certain tilt angle, L1 ≥ T*Lm, the tilt angle is determined to be an outlier. Here, T is a specified threshold, which can be determined according to the number of outliers to be removed, and is usually an integer greater than or equal to 2. In this embodiment, since the number of tilt angles is small, the number of outliers removed is generally 1. After removing the outliers, the remaining valid inclination angles are averaged (eg, arithmetic mean, weighted mean, etc.) to obtain an average angle. Generally, the average angle can be used as the inclination of the target palm.

[0114] By adopting the solution of the above embodiment, the tilt angle with large error can be eliminated, and the accuracy of the target palm tilt can be further improved, so that the final determined position frame is more accurate. At the same time, in the process of calculating the position frame, the four-finger pointing of the palm can also be obtained, which has a certain effect on gesture recognition in specific scenes.

[0115] It can be understood that if the slope value is used as a parameter to measure the inclination of the target palm, the average slope corresponding to the average angle can be used as the inclination of the target palm.

[0116] In an optional embodiment, step S400 includes the following steps:

[0117] S401, determining the direction of the target palm according to the inclination;

[0118] S402, if the direction of the target palm is vertical or horizontal, calculate the positions of four vertices of the initial detection frame according to the positions of the key points of the target palm, and determine the position frame according to the initial detection frame;

[0119] S403: If the direction of the target palm is not vertical or horizontal, calculate the functions corresponding to the four sides of the initial detection frame, and determine the position frame according to the initial detection frame.

[0120] In the above steps S401-S403, the direction of the target palm can be determined according to the inclination. For example, when the angle value is used as a parameter to measure the inclination, if the average angle is 90 degrees or 0 degrees, it means that the target palm is in the vertical direction or horizontal direction. If the average angle is other angles between 0 degrees and 90 degrees, it means that it is not in the vertical or horizontal direction. If the direction of the target palm is in the vertical or horizontal direction, the position frame can be a rectangular frame without inclination. At this time, the four vertex positions of the initial detection frame can be directly calculated according to the positions of each key point. If the direction of the target palm is not in the vertical or horizontal direction, at this time, it is necessary to calculate the functions corresponding to the four sides of the initial detection frame. These four sides at least pass through the positions of the top, bottom, rightmost and leftmost four key points. The slope is determined by the inclination of the target palm. After determining the four functions, the straight lines corresponding to the functions are determined at the same time. At this time, the four straight lines can be enclosed into a rectangular frame, i.e., the initial detection frame. The initial detection frame can generate a position frame, for example, the position frame can be obtained by magnifying it by a certain multiple, so as to avoid edge points being exactly on the boundary of the position frame, resulting in missing key points during gesture detection. In other embodiments, the initial detection frame can also be directly used as the position frame.

[0121] It should be noted that in this application, the vertical direction refers to the four fingers of the palm except the thumb are all vertically upward or vertically downward, and correspondingly, the base of the palm is in the vertical downward direction or vertical upward direction of the above four fingers. The horizontal direction refers to the four fingers of the palm except the thumb are horizontally left or horizontally right, and correspondingly, the base of the palm is in the horizontal right direction or horizontal left direction of the above four fingers. In this application, there is no restriction on the direction of the palm, which can be the palm facing the shooting direction or the palm facing away from the shooting direction.

[0122] In the above embodiment, step S401 includes:

[0123] S4011, when the inclination meets the first preset condition, determining whether the direction of the target palm is vertical or horizontal;

[0124] S4012, when the inclination does not meet the first preset condition, determine that the direction of the target palm is not vertical or horizontal, wherein the first preset condition is: the absolute value of the slope value corresponding to the inclination is greater than or equal to the first preset threshold value T1 or less than 1 / T1.

[0125] In the above steps S401-S403, the inclination can be an angle value or a slope value. If the angle value is determined, different judgment criteria need to be set for angles in different ranges (for example, the judgment criteria corresponding to 90 degrees and 270 degrees are different, but both belong to the vertical direction) to determine its direction. If the slope value is used for judgment, the judgment criteria can be unified by simply determining the absolute value. Therefore, in this embodiment, it is preferred to use the slope value for direction judgment. If the inclination is an average angle, the corresponding average slope is first obtained. If the inclination is an average slope, it is directly determined to be the slope value corresponding to the average slope. Assuming that the average slope is K, the direction of the target palm can be further corrected according to the value of the slope. When the absolute value of K is greater than or equal to the first preset threshold value T1 (for example, 10, or a larger value, which can be set as needed), it means that the inclination is close to 90 degrees or 270 degrees, and K=∞ is directly corrected, and it is determined that the direction of the target palm is vertical at this time. When the absolute value of K is less than or equal to 1 / T1, it can be considered that the tilt angle is close to 0 degrees or 180 degrees. K that meets this condition can determine that the target palm is in the horizontal direction, and directly correct K = 0. If K is greater than 1 / T1 and less than T2, no correction is required.

[0126] In actual application, the user's palm is usually not strictly vertical or horizontal. Therefore, a certain deviation can be allowed. As long as a certain inclination is reached, the palm can be considered to be vertical or horizontal. The method of the above embodiment is more in line with the actual application scenario. Based on the palm direction determined in this way, in the subsequent gesture recognition and tracking, it can avoid excessive sensitivity and misrecognition.

[0127] In the above embodiment, in step S402, the positions of four vertices of the initial detection frame are calculated according to the positions of the key points of the target palm, and the position frame is determined according to the initial detection frame, including:

[0128] S4021, taking the minimum horizontal coordinate value of all key points as the first horizontal coordinate value Xs, and the maximum horizontal coordinate value as the second horizontal coordinate value Xg;

[0129] S4022, taking the minimum ordinate value of all key points as the first ordinate value Ys, and the maximum ordinate value as the second ordinate value Yg;

[0130] S4023, using (Xs, Ys), (Xs, Yg), (Xg, Ys), (Xg, Yg) as the coordinates of each vertex of the initial detection frame;

[0131] S4024: Enlarge the initial detection frame to obtain the position frame.

[0132] In the above steps S4021-S4024, the initial detection frame and the position frame are both rectangular frames. When the target palm is in a vertical or horizontal direction, as long as the vertices determined in the above manner are used as the vertices of the initial detection frame, it can at least be ensured that the key points do not fall outside the initial detection frame. Considering that the 21 key points cannot cover the entire target palm, and some key points may just fall at the vertices or boundaries of the initial detection frame, the initial detection frame can be enlarged to a certain extent, for example: the magnification factor H is between 1.05 and 1.5. The specific method can be: the center of the initial detection frame remains unchanged, and the four sides move outward so that the area of ​​the rectangular frame is H times the area of ​​the initial detection frame.

[0133] In the above embodiment, in step S403, calculating the functions corresponding to the four sides of the initial detection frame, and determining the position frame according to the initial detection frame includes:

[0134] S4031, calculating a first maximum intercept Pg and a first minimum intercept Ps determined when a straight line with an average slope K passes through all key points of the target palm; the average slope K is determined according to the inclination of the target palm;

[0135] S4032, calculating the second maximum intercept Tg and the second minimum intercept Ts of a straight line having a slope of the negative reciprocal of the average slope K and passing through all key points of the target palm;

[0136] S4033, determining four functions corresponding to the four sides of the initial detection frame according to the first maximum intercept Pg, the first minimum intercept Ps, the second maximum intercept Tg, the second minimum intercept Ts and the average slope K, the four functions being: y=x*K+Ps, y=x*K+Pg, y=-x / K+Ts, y=-x / K+Tg;

[0137] S4034, using the rectangular frame surrounded by the four functions as the initial detection frame;

[0138] S4035, enlarging the initial detection frame to obtain the position frame.

[0139] In the above steps S4031-S4035, the rectangular frame surrounded by the four functions can refer to Figure 4 As shown, Figure 4 The following is an example of an embodiment of the present application. Figure 3Schematic diagram of the position frame generated by the target palm in the figure. If shown, p0 is the initial detection frame obtained by the above method. The four sides of the initial detection frame are determined by y=x*K+Ps, y=x*K+Pg, y=-x / K+Ts, and y=-x / K+Tg. The above four sides pass through the bottom, top, leftmost, and rightmost points of the 21 key points, respectively, to form a rectangular frame. The initial detection frame covers all 21 key points. After enlarging the initial detection frame in the above method, the final position frame p1 can be obtained.

[0140] In an optional embodiment, obtaining a position frame indicating the size and position of the target palm in step S400 includes:

[0141] Generate a rectangular frame covering all key points of the target palm;

[0142] The position frame is obtained by enlarging the rectangular frame.

[0143] In the above steps, the rectangular frame is equivalent to the initial detection frame as mentioned above. Considering that even if all key points are covered, it cannot cover the entire target palm, and some key points may just fall at the vertices or boundaries of the rectangular frame, the rectangular frame can be enlarged to a certain extent, for example: the enlargement factor H is between 1.05 and 1.5. The specific method can be: the center of the rectangular frame remains unchanged, and the sides are moved outward so that the area of ​​the position frame is H times the area of ​​the rectangular frame.

[0144] Please refer to Figure 5 , which is a functional module diagram of a palm position determination device provided in an embodiment of the present application, as shown in the figure, the palm position determination device 100 includes:

[0145] A key point detection module 110 is used to perform key point detection on the image to be detected containing the target palm, and determine each key point of the target palm;

[0146] A linear fitting module 120, configured to perform linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger, to obtain linear fitting lines corresponding to the multiple fingers;

[0147] An inclination determination module 130, configured to determine the inclination of the target palm according to the slopes of the plurality of linear fitting lines;

[0148] The position frame generating module 140 is used to obtain a position frame indicating the size and position of the target palm according to the inclination and the positions of the key points of the target palm, wherein the position frame covers all the key points of the target palm.

[0149] In this embodiment, the palm position determination device 100 may further include:

[0150] An image pre-positioning module is used to pre-position the target palm of the acquired image through a target detection and positioning model to obtain a pre-positioning result;

[0151] A sub-image acquisition module is used to crop a sub-image containing the target palm from the acquired image according to the pre-positioning result to obtain the image to be detected.

[0152] In an optional embodiment, the position frame generating module 140 generates a position frame covering all key points of the target palm including:

[0153] Generate a rectangular frame covering all key points of the target palm;

[0154] The position frame is obtained by enlarging the rectangular frame.

[0155] In an optional embodiment, the inclination determination module 130 includes:

[0156] An inclination angle determination unit 131, configured to determine a plurality of corresponding inclination angles according to the slopes of the plurality of linear fitting lines;

[0157] A validity judgment unit 132, configured to remove outliers from the plurality of tilt angles to obtain a plurality of valid tilt angles;

[0158] The inclination calculation unit 133 is used to calculate the average value of the multiple valid inclination angles to obtain an average angle, and determine the average angle as the inclination of the target palm.

[0159] In an optional embodiment, the position box generating module 140 includes:

[0160] A direction determining unit 141, configured to determine the direction of the target palm according to the inclination;

[0161] A position frame calculation unit 142 is used to calculate the positions of four vertices of the initial detection frame according to the positions of the key points of the target palm when the direction of the target palm is vertical or horizontal, and determine the position frame according to the initial detection frame; and

[0162] When the direction of the target palm is not vertical or horizontal, the functions corresponding to the four sides of the initial detection frame are calculated, and the position frame is determined according to the initial detection frame.

[0163] In the above embodiment, the direction determination unit 141 performs direction determination including:

[0164] When the inclination meets the first preset condition, determining that the direction of the target palm is vertical or horizontal;

[0165] When the inclination does not meet the first preset condition, it is determined that the direction of the target palm is not vertical or horizontal.

[0166] In the above embodiment, the position frame calculation unit 142 includes:

[0167] The coordinate calculation subunit 1421 is used for: when the direction of the target palm is vertical or horizontal:

[0168] The minimum horizontal coordinate value among all key points is taken as the first horizontal coordinate value Xs, and the maximum horizontal coordinate value is taken as the second horizontal coordinate value Xg;

[0169] The minimum ordinate value among all key points is taken as the first ordinate value Ys, and the maximum ordinate value is taken as the second ordinate value Yg;

[0170] Taking (Xs, Ys), (Xs, Yg), (Xg, Ys), (Xg, Yg) as the coordinates of each vertex of the initial detection frame;

[0171] The amplifying subunit 1422 is used for:

[0172] After the initial detection frame is enlarged, the position frame is obtained.

[0173] In the above embodiment, the position frame calculation unit 142 further includes:

[0174] The function obtaining unit 1423 is used for: when the direction of the target palm is not vertical or horizontal:

[0175] Calculate the first maximum intercept Pg and the first minimum intercept Ps determined when a straight line with an average slope K passes through all key points of the target palm; the average slope K is determined according to the inclination of the target palm;

[0176] Calculate the second maximum intercept Tg and the second minimum intercept Ts of a straight line with the negative reciprocal of the average slope K as the slope when the straight line passes through all key points of the target palm;

[0177] Determine four functions corresponding to the four sides of the initial detection frame according to the first maximum intercept Pg, the first minimum intercept Ps, the second maximum intercept Tg, the second minimum intercept Ts and the average slope K, the four functions being: y=x*K+Ps, y=x*K+Pg, y=-x / K+Ts, y=-x / K+Tg;

[0178] The rectangular frame enclosed by the four functions is used as the initial detection frame.

[0179] The functions of the palm position determination device 100 in the above embodiments can be used to execute the method described in the above method embodiments. The specific implementation thereof can refer to the description of the above method embodiments, and will not be described again for the sake of brevity.

[0180] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. If the functions are implemented in the form of software functional units and sold or used as independent products, they may be stored in a processor-executable non-volatile computer-readable storage medium.

[0181] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device 60 suitable for implementing the embodiment disclosed in the present application. The electronic device in the embodiment disclosed in the present application may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments disclosed in this application.

[0182] like Figure 6 As shown, the electronic device 60 may include a processor (e.g., a central processing unit, a graphics processor, etc.) 601, which may perform various appropriate actions and processes according to different application codes stored in the memory 601. The memory 601 may include a random access memory (RAM), a read-only memory (ROM). In the RAM 603, various programs and data required for the operation of the electronic device 60 are stored. The memory 601, the processor 602, and the communication interface 603 may be connected by lines, for example, connected to each other through a bus 604. The communication interface 603 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data.

[0183] In the embodiment of the present application, when the processor 602 executes the application code stored in the memory 601, the following can be performed: Figure 1 The method for determining the palm position provided in the embodiment.

[0184] Although the electronic device 60 having various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0185] In particular, according to the embodiments disclosed in the present application, the above reference Figure 1 The process described in the flowchart of the embodiment can be implemented as a computer program. For example, the embodiment of the present application also provides a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a method for executing Figure 1 The program code of the method of the embodiment flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication interface 603, or installed from the memory 608. When the computer program is executed by the processor 602, the above functions defined in the method of the above embodiment are performed.

[0186] It should be noted that the above-mentioned computer-readable medium may include, but is not limited to: may include volatile memory (volatile memory), such as random access memory (random access memory, RAM); memory 802 may also include non-volatile memory (non-volatile memory), such as read-only memory (read-only memory, ROM), flash memory (flash memory), hard disk drive (hard disk drive, HDD) or solid-state drive (solid-state drive, SSD); memory 802 may also include a combination of the above-mentioned types of memory

[0187] In the present application, a computer readable storage medium may be any tangible medium that can contain or store a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0188] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0189] In an optional embodiment, the present application also provides a computer storage medium that can be used for computer software instructions, including a computer program, and when the computer program is executed by a processor, the palm position determination method described in the above embodiment is executed. The storage medium includes but is not limited to a flash memory, a hard disk, and a solid state drive.

[0190] Those skilled in the art will appreciate that, without conflict, the above-mentioned preferred solutions can be freely combined and superimposed.

[0191] It should be understood that the above-mentioned embodiments are merely illustrative and not restrictive. Without departing from the basic principles of the present application, various obvious or equivalent modifications or substitutions that can be made by technicians in this field to the above details will be included in the scope of the claims of the present application.

Claims

1. A palm position determination method, characterized in that: The method comprises: Performing key point detection on the image to be detected containing the target palm to determine each key point of the target palm; Performing linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger to obtain linear fitting lines corresponding to the multiple fingers; Determining the inclination of the target palm according to the slopes of the plurality of linear fitting lines; Determining the direction of the target palm according to the inclination; If the direction of the target palm is vertical or horizontal, the positions of four vertices of the initial detection frame are calculated according to the positions of the key points of the target palm, and the position frame is determined according to the initial detection frame; If the direction of the target palm is not vertical or horizontal, calculate the first maximum intercept Pg and the first minimum intercept Ps determined when a straight line with an average slope K passes through all key points of the target palm; the average slope K is determined according to the inclination of the target palm; calculate the second maximum intercept Tg and the second minimum intercept Ts when a straight line with a negative reciprocal of the average slope K passes through all key points of the target palm; determine four functions corresponding to the four sides of the initial detection frame according to the first maximum intercept Pg, the first minimum intercept Ps, the second maximum intercept Tg, the second minimum intercept Ts and the average slope K, the four functions are: y=x*K+Ps, y=x*K+Pg, y=-x / K+Ts, y=-x / K+Tg; use the rectangular frame surrounded by the four functions as the initial detection frame; after enlarging the initial detection frame, obtain the position frame, The position frame covers all key points of the target palm.

2. The palm position determination method according to claim 1, characterized in that: Before performing key point detection on the image to be detected containing the target palm, the method further includes: Pre-position the target palm of the acquired image through the target detection and positioning model to obtain a pre-positioning result; A sub-image including the target palm is cropped from the acquired image according to the pre-positioning result to obtain the image to be detected.

3. The palm position determination method according to claim 1, characterized in that: Determining the inclination of the target palm according to the slopes of the plurality of linear fitting lines comprises: Determining a plurality of corresponding inclination angles according to the slopes of the plurality of linear fitting straight lines; After removing outliers from the plurality of tilt angles, a plurality of valid tilt angles are obtained; An average value is calculated for the multiple effective inclination angles to obtain an average angle, and the average angle is determined to be the inclination of the target palm.

4. The palm position determination method according to claim 1, characterized in that: Determining the direction of the target palm according to the inclination includes: When the inclination meets the first preset condition, determining that the direction of the target palm is vertical or horizontal; When the inclination does not meet the first preset condition, it is determined that the direction of the target palm is not vertical or horizontal, wherein the first preset condition is: the absolute value of the slope value corresponding to the inclination is greater than or equal to the first preset threshold T1 or less than 1 / T1.

5. The palm position determination method according to claim 1, characterized in that: The calculating the positions of four vertices of the initial detection frame according to the positions of the key points of the target palm, and determining the position frame according to the initial detection frame comprises: The minimum horizontal coordinate value among all key points is taken as the first horizontal coordinate value Xs, and the maximum horizontal coordinate value is taken as the second horizontal coordinate value Xg; The minimum ordinate value among all key points is taken as the first ordinate value Ys, and the maximum ordinate value is taken as the second ordinate value Yg; Taking (Xs, Ys), (Xs, Yg), (Xg, Ys), (Xg, Yg) as the coordinates of each vertex of the initial detection frame; After the initial detection frame is enlarged, the position frame is obtained.

6. The palm position determination method according to claim 1, characterized in that: Obtaining a position frame indicating the size and position of the target palm includes: Generate a rectangular box covering all key points of the target palm; The position frame is obtained by enlarging the rectangular frame.

7. A palm position determination device, characterized in that: The palm position determining device comprises: A key point detection module is used to perform key point detection on the image to be detected containing the target palm, and determine each key point of the target palm; A linear fitting module, used for performing linear fitting on the key points of the base of the target palm and multiple key points belonging to the same finger, to obtain linear fitting lines corresponding to multiple fingers; An inclination determination module, used to determine the inclination of the target palm according to the slopes of the plurality of linear fitting lines; A position frame generating module, used for obtaining a position frame indicating the size and position of the target palm according to the inclination and the positions of the key points of the target palm, wherein the position frame covers all the key points of the target palm; wherein, according to the inclination, the direction of the target palm is determined; If the direction of the target palm is vertical or horizontal, the positions of four vertices of the initial detection frame are calculated according to the positions of the key points of the target palm, and the position frame is determined according to the initial detection frame; If the direction of the target palm is not vertical or horizontal, calculate the first maximum intercept Pg and the first minimum intercept Ps determined when a straight line with an average slope K passes through all key points of the target palm; the average slope K is determined according to the inclination of the target palm; calculate the second maximum intercept Tg and the second minimum intercept Ts when a straight line with a negative reciprocal of the average slope K passes through all key points of the target palm; determine four functions corresponding to the four sides of the initial detection frame according to the first maximum intercept Pg, the first minimum intercept Ps, the second maximum intercept Tg, the second minimum intercept Ts and the average slope K, and the four functions are: y=x*K+Ps, y=x*K+Pg, y=-x / K+Ts, y=-x / K+Tg; use the rectangular frame enclosed by the four functions as the initial detection frame; after enlarging the initial detection frame, obtain the position frame.

8. An electronic device, comprising a processor, characterized in that: The processor is used to implement the palm position determination method according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the palm position determination method according to any one of claims 1 to 6 is executed.

Images