Hand image processing method, device and equipment

Abstract

The invention discloses a hand image processing method, device and equipment. After the hand image is obtained, the posture information of the target finger in the hand image is recognized, and the shielding degree parameter of the target finger can be determined based on the posture information of the target finger. And if the shielding degree parameter of the target finger is within a preset parameter range, determining that the target finger is shielded. Whether the target finger in the hand image is shielded or not is identified, so that a more complex use scene can be met.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a hand image processing method, device and equipment. Background Art

[0002] With the booming development of Internet applications, various applications have scenarios where users interact with their hands. In the use of such scenarios, it is necessary to process the input hand image, such as identifying whether there is a hand area in the input image. However, the current processing of hand images is relatively simple and cannot meet more complex usage scenarios. Summary of the invention

[0003] In view of this, embodiments of the present application provide a hand image processing method, apparatus, and device to identify whether a target finger in a hand image is blocked.

[0004] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a hand image processing method, the method comprising:

[0006] Acquire a hand image;

[0007] Identifying the posture information of the target finger in the hand image;

[0008] Determining an occlusion degree parameter of the target finger according to the posture information of the target finger;

[0009] If the occlusion degree parameter of the target finger is within a preset parameter range, it is determined that the target finger is occluded.

[0010] In a second aspect, an embodiment of the present application provides a hand image processing device, wherein the method and device:

[0011] An acquisition unit, used for acquiring a hand image;

[0012] A recognition unit, used to recognize the posture information of the target finger in the hand image;

[0013] A first determining unit, configured to determine an occlusion degree parameter of the target finger according to the posture information of the target finger;

[0014] The second determining unit is configured to determine that the target finger is blocked if the blocking degree parameter of the target finger is within a preset parameter range.

[0015] In a third aspect, an embodiment of the present application provides a hand image processing device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the hand image processing method as described above is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the hand image processing method as described above.

[0017] It can be seen that the embodiments of the present application have the following beneficial effects:

[0018] After acquiring a hand image, the embodiment of the present application identifies the posture information of the target finger in the hand image. Based on the posture information of the target finger, the occlusion degree parameter of the target finger can be determined. If the occlusion degree parameter of the target finger is within a preset parameter range, it is determined that the target finger is occluded. The recognition of whether the target finger in the hand image is occluded is achieved, thereby meeting more complex usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of an exemplary application scenario provided for an embodiment of the present application;

[0020] Figure 2 A flowchart of a hand image processing method provided in an embodiment of the present application;

[0021] Figure 3 A flowchart of another hand image processing method provided in an embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of the spin angle of 0 degrees in the embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of an embodiment of the present application in which the spin angle is 90 degrees;

[0024] Figure 6 This is a schematic diagram of a spin angle of 270 degrees in an embodiment of the present application;

[0025] Figure 7 A flowchart of another hand image processing method provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of the position of the identification frame in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of another application scenario provided by an embodiment of the present application;

[0028] Fig.10 A schematic diagram of a hand image processing device provided in an embodiment of the present application;

[0029] Fig.11 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] In order to facilitate understanding and explanation of the technical solution provided by the embodiments of the present application, the background technology of the embodiments of the present application will be described below.

[0032] With the vigorous development of Internet applications, various applications have scenarios of interacting with user hands. For example, in an AR (Augment Reality) hand jewelry try-on application, the hand jewelry try-on effect can be displayed based on a hand image.

[0033] In the use of such scenarios, it is usually necessary to process the input hand image. However, the current processing of hand images is relatively simple and cannot meet more complex usage scenarios. For example, in the application of AR hand jewelry try-on, currently only the hand image can be used to give a prompt whether there is a hand area in the image, but due to the complexity of hand postures, it is impossible to avoid abnormal try-on effects caused by complex hand postures.

[0034] Based on this, the embodiments of the present application provide a hand image processing method, device and apparatus, which can identify whether the target finger in the hand image is blocked based on the posture information of the target finger in the hand image. Thus, more complex usage scenarios can be met. For example, if applied to the scene of AR hand jewelry trying on, when it is identified that the target finger is blocked, the user can be guided to make the correct gesture to avoid some abnormal try-on effects.

[0035] In order to facilitate understanding of the hand image processing method provided in the embodiment of the present application, Figure 1 See the example scenario shown in the figure. Figure 1 As shown, this figure is a schematic diagram of an exemplary application scenario provided in an embodiment of the present application.

[0036] The hand image processing method provided in the embodiment of the present application can be applied to a client that performs hand image processing. For example, in practical applications, it can be applied to a client that implements AR hand jewelry try-on. In the embodiment of the present application, a hand image is first obtained, and the posture information of the target finger in the hand image, such as the 6D posture, is identified. The occlusion degree parameter of the target finger can be determined based on the posture information of the target finger. If the occlusion degree parameter of the target finger is within the preset parameter range, it is determined that the target finger is occluded. It is realized to identify whether there is occlusion of the target finger in the hand image, so as to meet more complex usage scenarios. For example, when applied to the scene of implementing AR hand jewelry try-on, the target finger is the finger that mounts the hand jewelry (which can be referred to as the mounting finger), so as to realize the identification of whether there is occlusion of the mounting finger. When it is identified that the mounting finger is occluded, the user can be guided to make the correct gesture to avoid abnormal try-on effects.

[0037] Those skilled in the art will understand that Figure 1 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0038] To facilitate understanding of the embodiments of the present application, a hand image processing method provided in the embodiments of the present application is described below with reference to the accompanying drawings.

[0039] See also Figure 2 As shown in FIG. 1 , this figure is a flow chart of a hand image processing method provided in an embodiment of the present application, as shown in FIG. Figure 2 As shown, the method may include S201-S204:

[0040] S201: Acquire a hand image.

[0041] In order to process the hand image, it is first necessary to obtain the hand image, which is generally an image including the user's hand area, for example, an image including the user's single hand area. In practical applications, the hand image uploaded by the user to the client can be obtained, or the hand image collected in real time by the camera module of the device where the client is located can be obtained. The embodiment of the present application does not limit the specific implementation method of obtaining the hand image.

[0042] S202: Identify the posture information of the target finger in the hand image.

[0043] In the hand image, there are usually multiple fingers of a user's hand, so the posture information of the target finger in the hand image can be identified. The target finger can be a finger that needs to be further processed in the hand image. For example, in the scenario of implementing AR hand jewelry try-on, the target finger is the finger that mounts the hand jewelry, that is, the mounting finger. The user can specify the target finger, for example, specify the middle finger as the target finger. In actual applications, the target finger can be first identified from the hand image, and then the posture information of the target finger can be further identified.

[0044] The pose information is used to characterize the pose of the target finger in space. For example, the pose information can be a 6D pose. 6D pose refers to the position and orientation of an object in three-dimensional space, which is usually described by six degrees of freedom. These six degrees of freedom include three translational degrees of freedom and three rotational degrees of freedom of the object in three-dimensional space. Then the pose information of the target finger includes three translational degrees of freedom of the translation part and three degrees of freedom of the rotation part. In practical applications, the three degrees of freedom of the rotation part in the pose information can be represented by a 3×3 rotation matrix.

[0045] The identification of the target finger's posture information can be implemented using an end-to-end method based on a neural network or a method based on key points. The embodiment of the present application does not limit the specific implementation method of identifying the target finger's posture information.

[0046] S203: Determine an occlusion degree parameter of the target finger according to the posture information of the target finger.

[0047] In an embodiment of the present application, an occlusion degree parameter of the target finger can be determined based on the posture information of the target finger, and the occlusion degree parameter of the target finger characterizes the occlusion degree of the target finger. In a possible implementation, the occlusion degree parameter of the target finger includes the spin angle of the target finger and / or the probability of overlap between the target finger and an adjacent finger. That is, whether the target finger is occluded can be determined by the spin angle of the target finger, and / or whether the target finger is occluded can be determined by the degree of overlap between the target finger and an adjacent finger. Possible implementation methods for determining the occlusion degree parameter of the target finger will be described in detail in subsequent embodiments and will not be elaborated here.

[0048] S204: If the occlusion degree parameter of the target finger is within a preset parameter range, it is determined that the target finger is occluded.

[0049] If the occlusion degree parameter of the target finger is within the preset parameter range, it means that the occlusion degree of the target finger is high. At this time, the target finger is blocked, which is not conducive to various applications to further process the target finger. In a possible implementation, when the occlusion degree parameter includes the spin angle of the target finger, the preset parameter range can be a preset angle range, and when the occlusion degree parameter includes the probability of overlap between the target finger and the adjacent fingers, the preset parameter range can be a preset probability range. The value of the preset parameter range can be set according to the actual situation, and the embodiment of the present application does not limit the value of the preset parameter range.

[0050] Based on the description of S201-S204, after acquiring the hand image, the embodiment of the present application identifies the posture information of the target finger in the hand image. Based on the posture information of the target finger, the occlusion degree parameter of the target finger can be determined. If the occlusion degree parameter of the target finger is within the preset parameter range, it is determined that the target finger is occluded. It is possible to identify whether the target finger in the hand image is occluded, thereby meeting more complex usage scenarios.

[0051] The following describes the implementation of the hand image processing method provided by the embodiment of the present application when the occlusion degree parameter includes the rotation angle of the target finger. Figure 3 As shown in FIG. 1 , this figure is a flow chart of another hand image processing method provided by an embodiment of the present application, as shown in FIG. Figure 3 As shown, the method may include S301-S304:

[0052] S301: Acquire a hand image.

[0053] S302: Identify the posture information of the target finger in the hand image.

[0054] The implementation of S301 - S302 is the same as that of S201 - S202 . For related descriptions, please refer to the above embodiments and will not be repeated here.

[0055] S303: Determine the spin angle of the target finger according to the posture information of the target finger.

[0056] S303 is a possible implementation method of S203 for determining the occlusion degree parameter of the target finger according to the posture information of the target finger.

[0057] In the embodiment of the present application, the hand rotation angle can be used to determine the degree of occlusion of the hand area. The hand rotation angle is the rotation angle along the fingertip to wrist axis. In specific implementation, the hand rotation angle can be replaced by the target finger rotation angle. Thus, the target finger rotation angle is used as the occlusion degree parameter, and the target finger posture information is used to determine the target finger rotation angle.

[0058] The rotation part in the target finger's posture information can be decomposed into the target finger's spin and other rotations except the spin. Since the target finger's spin angle cannot be directly calculated, the rotation matrix of the target finger's other rotations can be calculated from the rotation part in the target finger's posture information, and then the rotation matrix of the target finger's spin can be derived, and finally the target finger's spin angle can be obtained.

[0059] In a possible implementation, the specific implementation of S303 determining according to the posture information of the target finger may include A1-A3:

[0060] A1: According to the posture information of the target finger, obtain the rotation matrix of other rotations of the target finger.

[0061] In the embodiment of the present application, the rotation matrix of the target finger in the posture information can be first calculated to calculate the rotation matrix of other rotations of the target finger. The rotation matrix in the posture information of the target finger is generally a 3×3 matrix, and the rotation matrix of other rotations of the target finger is generally also a 3×3 matrix.

[0062] In a possible implementation, A1 may obtain the rotation matrix of other rotations of the target finger according to the posture information of the target finger, and the specific implementation may include B1-B5:

[0063] B1: Perform matrix multiplication on the rotation matrix in the posture information of the target finger and the vector of the initial spin axis of the target finger to obtain the vector of the rotated spin axis of the target finger.

[0064] The initial spin axis of the target finger can be the spin axis of the occluding object of the target finger, and the direction is from the base to the fingertip. When processing the hand image, the occluding objects of each finger can be predefined. The occluding object of the finger is generally defined as a cylinder, which can be used to simulate the initial state of the finger. The vector of the initial spin axis of the target finger is a 3×1 vector. The rotation matrix in the posture information of the target finger is matrix-multiplied with the vector of the initial spin axis of the target finger, which is equivalent to the rotation matrix in the posture information acting on the initial spin axis of the target finger, so that the initial spin axis of the target finger rotates, and the vector of the rotated spin axis of the target finger is obtained. The formula can be expressed as: rv=R*v. Among them, rv represents the vector of the rotated spin axis of the target finger, R represents the rotation matrix in the posture information, v represents the vector of the initial spin axis of the target finger, and * represents matrix multiplication.

[0065] B2: Perform a cross product of the vector of the rotated spin axis of the target finger and the vector of the initial spin axis of the target finger to obtain a cross product result.

[0066] By cross-producting the vector of the target finger's rotated spin axis with the vector of the target finger's initial spin axis, the situation of the rotation axis of other rotations can be obtained. The direction of the cross-product result is perpendicular to the plane formed by the vector of the target finger's rotated spin axis and the vector of the target finger's initial spin axis, and the magnitude is the square of the sine value of the target angle, which is the angle between the target finger's initial spin axis and the target finger's rotated spin axis.

[0067] The formula of this step can be expressed as: cv = v × rv, where cv represents the cross product result, and × represents the cross product between vectors.

[0068] B3: Calculate the square of the cross product result to obtain the square of the sine value of the target angle; the target angle is the angle between the initial spin axis of the target finger and the spin axis of the target finger after rotation.

[0069] Based on the above description, the size of the cross product result is the square of the sine value of the target angle, that is, sin 2 θ=|cv| 2 , where θ is the target angle.

[0070] B4: Perform a dot multiplication of the vector of the rotated spin axis of the target finger and the vector of the initial spin axis of the target finger to obtain the cosine value of the target angle.

[0071] The cosine value of the target angle is obtained by performing a dot product between the vector of the target finger's rotated spin axis and the vector of the target finger's initial spin axis. The formula can be expressed as cosθ=v·rv, where · represents a dot product between vectors.

[0072] B5: According to the square of the sine value of the target angle, the cosine value of the target angle and the cross product result, the rotation matrix of other rotations of the target finger is obtained.

[0073] The square of the sine value of the target angle and the cosine value of the target angle are parameters related to the rotation angles of other rotations. The cross product result is a parameter related to the rotation axis of other rotations. Based on the above parameters, the rotation matrix of other rotations of the target finger can be obtained. The formula can be expressed as prec rot =f(cosθ,sin 2 θ,cv), where prec rot The rotation matrix representing other rotations of the target finger, f is a method for solving the rotation matrix based on parameters related to the rotation axis and parameters related to the rotation angle.

[0074] A2: According to the rotation matrix of other rotations of the target finger and the rotation matrix in the posture information of the target finger, the rotation matrix of the rotation axis of the target finger is obtained.

[0075] Since the rotation matrix in the posture information of the target finger can be decomposed into the rotation matrix of other rotations of the target finger and the rotation matrix of the spin axis after the rotation of the target finger, when R and prec are known rot In the case of , the rotation matrix spin of the target finger after rotation can be solved rot , which is a 3×3 matrix. The formula for this step can be expressed as: spin rot =prec rot T *R.

[0076] A3: According to the rotation matrix of the spin axis of the target finger after rotation, the spin angle of the target finger is obtained.

[0077] Finally, the rotation matrix spin of the target finger after rotation is obtained rot , the spin angle of the target finger can be deduced.

[0078] In this way, the embodiment of the present application can determine the spin angle of the target finger through the posture information of the target finger.

[0079] S304: If the rotation angle of the target finger is within a preset angle range, it is determined that the target finger is blocked.

[0080] S304 is a possible implementation method of determining that the target finger is blocked if the blocking degree parameter of the target finger is within a preset parameter range in S204.

[0081] If the rotation angle of the target finger is within the preset angle range, it can be determined that the target finger is blocked. The preset angle range can be set according to actual conditions. For example, see Figure 4 As shown, assuming that the back of the right hand is 0 degrees, counterclockwise rotation is positive, when rotated to 90 degrees, see Figure 5 As shown, or when rotated to 270 degrees, see Figure 6 As shown, two sides of the hand can be obtained, at which the degree of occlusion of the fingers is the greatest. Based on this, combined with actual experiments, 60-120 degrees and 240-300 degrees can be set as preset angle ranges, indicating two angle intervals where the fingers are severely occluded.

[0082] Based on the description of S301-S304, after acquiring the hand image, the embodiment of the present application identifies the posture information of the target finger in the hand image. Based on the posture information of the target finger, the spin angle of the target finger can be determined. If the spin angle of the target finger is within the preset angle range, it is determined that the target finger is blocked. Through the spin angle of the target finger, it is possible to identify whether the target finger in the hand image is blocked, thereby meeting more complex usage scenarios.

[0083] The occlusion of the target finger can be estimated by the rotation angle of the target finger. However, the finger has a high degree of freedom, and the rotation angle of the target finger may not accurately predict the occlusion of the target finger under various gestures. Therefore, the embodiment of the present application also proposes to judge the occlusion of the target finger by the position of the finger's recognition frame. That is, when the occlusion degree parameter includes the probability of overlap between the target finger and the adjacent fingers, the implementation method of the hand image processing method provided by the embodiment of the present application is described. See Figure 7 As shown in FIG. 1 , this figure is a flow chart of another hand image processing method provided by an embodiment of the present application, as shown in FIG. Figure 7 As shown, the method may include S701-S707:

[0084] S701: Acquire a hand image.

[0085] S702: Identify the posture information of the target finger in the hand image.

[0086] The implementation of S701-S702 is the same as that of S201-S202. For related descriptions, please refer to the above embodiment and will not be repeated here.

[0087] S703: Identify the posture information of the fingers adjacent to the target finger in the hand image.

[0088] While identifying the posture information of the target finger, the posture information of the adjacent fingers of the target finger can also be identified. It is understandable that the adjacent fingers of the target finger can have two fingers, including a first adjacent finger and a second adjacent finger. For example, if the target finger is the middle finger, the adjacent fingers of the target finger are the index finger and the ring finger. In practical applications, the adjacent fingers of the target finger can be first identified from the hand image, and then the posture information of the adjacent fingers of the target finger can be further identified.

[0089] The specific implementation of S203 determining the occlusion degree parameter of the target finger according to the posture information of the target finger includes the following S704-S706.

[0090] S704: Determine the position of the recognition frame of the target finger according to the posture information of the target finger.

[0091] By using the posture information of the target finger, the predefined occluding object of the target finger and the camera intrinsic parameters, the occluding object of the target finger can be rendered on the hand image to correspond to the target finger, and the recognition frame position of the target finger can be obtained.

[0092] S705: Determine the position of the recognition frame of the adjacent finger according to the posture information of the adjacent finger.

[0093] Similarly, using the posture information of adjacent fingers, the predefined occlusion objects of adjacent fingers and the camera intrinsic parameters, the occlusion objects of adjacent fingers can be rendered on the hand image to correspond to the adjacent fingers, and the recognition frame positions of the adjacent fingers can be obtained. Figure 8 As shown, taking the target finger as the middle finger as an example, the white rectangle in the figure is the recognition frame position of the target finger and the adjacent fingers.

[0094] S706: Determine the probability of the target finger and the adjacent finger overlapping each other based on the position of the recognition frame of the target finger and the position of the recognition frame of the adjacent finger.

[0095] The probability of the target finger overlapping with the adjacent fingers can be determined by the intersection of the target finger's identification frame position and the identification frame positions of the left and right adjacent fingers. The probability of the target finger overlapping with the adjacent fingers can represent the degree of overlap between the target finger and the adjacent fingers. If the degree of overlap is high, it means that the target finger is blocked.

[0096] In a possible implementation, the neighboring fingers of the target finger include a first neighboring finger and a second neighboring finger. S706 determines the probability of the target finger and the neighboring finger overlapping each other based on the recognition frame position of the target finger and the recognition frame position of the neighboring finger. The specific implementation may include C1-C4:

[0097] C1: Determine the overlapping area of ​​the target finger and the first adjacent finger based on the identification frame position of the target finger and the identification frame position of the first adjacent finger.

[0098] In practical applications, the recognition frame position of the target finger and the recognition frame position of the first adjacent finger can both be represented by a two-dimensional matrix, and the intersection of the recognition frame position of the target finger and the recognition frame position of the first adjacent finger can be obtained to determine the overlapping area of ​​the target finger and the first adjacent finger.

[0099] That is, inter_l = mask_l⊙mask_f, where inter_l represents the overlapping area between the target finger and the first adjacent finger, which is an M×N matrix, M and N are the length and width of the hand image, mask_l represents the position of the recognition frame of the first adjacent finger, which is an M×N matrix, mask_f represents the position of the recognition frame of the target finger, which is an M×N matrix, and ⊙ represents the intersection. ∑inter_l is the overlapping area between the target finger and the first adjacent finger.

[0100] C2: Determine the overlapping area of ​​the target finger and the second adjacent finger based on the identification frame position of the target finger and the identification frame position of the second adjacent finger.

[0101] Similarly, the recognition frame position of the second adjacent finger can be represented by a two-dimensional matrix, and the intersection of the recognition frame position of the target finger and the recognition frame position of the second adjacent finger can be obtained to determine the overlapping area of ​​the target finger and the second adjacent finger.

[0102] That is, inter_r = mask_r⊙mask_f, where inter_r represents the overlapping area between the target finger and the second adjacent finger, which is an M×N matrix, M and N are the length and width of the hand image, mask_r represents the position of the recognition frame of the second adjacent finger, which is an M×N matrix, mask_f represents the position of the recognition frame of the target finger, which is an M×N matrix, and ⊙ represents the intersection. ∑inter_r is the overlapping area between the target finger and the second adjacent finger.

[0103] C3: Calculate the ratio of the overlapping area between the target finger and the first adjacent finger to the area of ​​the target finger to obtain the overlapping probability between the target finger and the first adjacent finger; the area of ​​the target finger is determined by the position of the recognition frame of the target finger.

[0104] The probability of the target finger and the first adjacent finger overlapping each other is obtained by calculating the ratio of the overlapping area of ​​the target finger and the first adjacent finger to the area of ​​the target finger. The area of ​​the target finger is determined by the position of the identification frame of the target finger, for example, the area of ​​the minimum circumscribed frame of the target finger.

[0105] Right now, Among them, ratio_l represents the probability of overlap between the target finger and the first adjacent finger, and bbox_f represents the area of ​​the target finger.

[0106] C4: Calculate the ratio of the overlapping area between the target finger and the second adjacent finger to the area of ​​the target finger to obtain the overlapping probability between the target finger and the second adjacent finger.

[0107] Similarly, this step can be expressed as Wherein, ratio_r represents the probability of overlap between the target finger and the second adjacent finger.

[0108] S707: If the probability of the target finger overlapping with the adjacent fingers is within a preset probability range, it is determined that the target finger is blocked.

[0109] S707 is a specific implementation of S204, if the occlusion degree parameter of the target finger is within a preset parameter range, determining that the target finger is occluded.

[0110] If the probability of the target finger overlapping with the adjacent fingers is within a preset probability range, it can be determined that the target finger is blocked. The preset probability range can be set according to actual conditions.

[0111] In a possible implementation, if the probability of the target finger and the adjacent finger overlapping each other is within a preset probability range, the specific implementation of determining that the target finger is blocked may include:

[0112] If the probability of the target finger overlapping with the first adjacent finger or the probability of the target finger overlapping with the second adjacent finger is within a preset probability range, it is determined that the target finger is blocked.

[0113] That is, if ratio_1 or ratio_r is greater than the preset probability threshold, it can be determined that the target finger is blocked, and the preset probability threshold can be set according to actual conditions.

[0114] Based on the description of S701-S707, after acquiring the hand image, the embodiment of the present application identifies the posture information of the target finger and the posture information of the adjacent fingers in the hand image. Based on the posture information of the target finger and the posture information of the adjacent fingers, the probability of the target finger and the adjacent fingers overlapping can be determined. If the probability of the target finger and the adjacent fingers overlapping is within a preset probability range, it is determined that the target finger is obstructed. Through the probability of the target finger and the adjacent fingers overlapping, it is possible to identify whether the target finger in the hand image is obstructed, thereby meeting more complex usage scenarios.

[0115] Based on the above embodiments, in a possible implementation, before identifying the posture information of the target finger in the hand image, the hand image processing method provided in the embodiment of the present application may further include:

[0116] Identify whether there is a hand region in the hand image.

[0117] If the hand region exists in the hand image, the posture information of the target finger in the hand image is recognized.

[0118] If the hand region does not exist in the hand image, a first prompt page is generated, and the first prompt page is used to display first prompt information, and the first prompt information is used to prompt that the hand region does not exist in the hand image.

[0119] That is, after acquiring the hand image, first identify whether there is a hand area in the hand image. If there is no hand area, display the first prompt page, which is used to display the first prompt information indicating that there is no hand area in the hand image, so as to guide the user to put the hand area into the picture. If there is a hand area, continue to determine whether the target finger is blocked. In practical applications, any lightweight detection algorithm can be used to identify whether there is a hand area in the hand image. The embodiment of the present application does not limit the specific implementation method of identifying whether there is a hand area in the hand image.

[0120] In a possible implementation, the hand image processing method provided in the embodiment of the present application may further include:

[0121] If it is determined that the target finger is blocked, a second prompt page is generated, the second prompt page is used to display second prompt information, and the second prompt information is used to prompt that the target finger is blocked.

[0122] If it is determined that the target finger is blocked, a second prompt page is generated, which is used to display the second prompt information indicating that the target finger is blocked, so as to guide the user to make a correct gesture. If the target finger is not blocked, the target finger can be further processed. For example, in an AR hand jewelry try-on application, the hand jewelry can be rendered normally on the target finger.

[0123] See also Fig. 9 As shown, a schematic diagram of another application scenario provided by an embodiment of the present application is shown.

[0124] After acquiring the hand image, it is identified whether there is a hand area in the hand image. If there is no hand area, a first prompt page is generated and displayed. If there is a hand area, the posture information of the target finger in the hand image is identified, and the occlusion degree parameter of the target finger is determined according to the posture information of the target finger. According to the occlusion degree parameter of the target finger, it is determined whether the target finger is occluded. If the target finger is occluded, a second prompt page is generated and displayed. If the target finger is not occluded, the target finger can be further processed, such as rendering hand jewelry normally on the target finger.

[0125] In this way, in an embodiment of the present application, when there is no hand area in the hand image, a first prompt page can be displayed, and when the target finger is obstructed, a second prompt page can be displayed, thereby guiding the user to adjust the hand posture through the front-end page.

[0126] Based on a hand image processing method provided in the above method embodiment, an embodiment of the present application also provides a hand image processing device, which will be described below in conjunction with the accompanying drawings.

[0127] See also Fig.10 As shown in FIG. 1 , this figure is a schematic diagram of the structure of a hand image processing device provided in an embodiment of the present application. Fig.10 As shown, the hand image processing device comprises:

[0128] An acquisition unit 1001 is used to acquire a hand image;

[0129] A first recognition unit 1002, configured to recognize the gesture information of the target finger in the hand image;

[0130] A first determining unit 1003 is used to determine an occlusion degree parameter of the target finger according to the posture information of the target finger;

[0131] The second determining unit 1004 is configured to determine that the target finger is blocked if the blocking degree parameter of the target finger is within a preset parameter range.

[0132] In a possible implementation manner, the occlusion degree parameter of the target finger includes a rotation angle of the target finger and / or a probability of overlap between the target finger and an adjacent finger.

[0133] In a possible implementation manner, when the occlusion degree of the target finger includes a spin angle of the target finger, the first determining unit includes:

[0134] A first determining subunit, configured to determine a spin angle of the target finger according to the posture information of the target finger;

[0135] The second determining unit is specifically configured to:

[0136] If the rotation angle of the target finger is within a preset angle range, it is determined that the target finger is blocked.

[0137] In a possible implementation manner, when the occlusion degree of the target finger includes the probability of overlap between the target finger and an adjacent finger, the first recognition unit is further configured to:

[0138] Identifying gesture information of fingers adjacent to the target finger in the hand image;

[0139] The first determining unit includes:

[0140] A second determining subunit, configured to determine a recognition frame position of the target finger according to the posture information of the target finger;

[0141] A third determining subunit, configured to determine a position of a recognition frame of the adjacent finger according to the posture information of the adjacent finger;

[0142] a fourth determining subunit, configured to determine a probability of overlap between the target finger and the adjacent finger based on the position of the recognition frame of the target finger and the position of the recognition frame of the adjacent finger;

[0143] The second determining unit is specifically configured to:

[0144] If the probability of the target finger overlapping the adjacent finger is within a preset probability range, it is determined that the target finger is blocked.

[0145] In a possible implementation manner, the first determining subunit includes:

[0146] a fifth determining subunit, configured to obtain a rotation matrix of other rotations of the target finger according to the posture information of the target finger;

[0147] a sixth determining subunit, configured to obtain a rotation matrix of the rotated spin axis of the target finger according to other rotational rotation matrices of the target finger and the rotation matrix in the posture information of the target finger;

[0148] The seventh determination subunit is used to obtain the spin angle of the target finger according to the rotation matrix of the spin axis of the target finger after rotation.

[0149] In a possible implementation manner, the fifth determining subunit is specifically configured to:

[0150] Performing matrix multiplication of the rotation matrix in the posture information of the target finger and the vector of the initial spin axis of the target finger to obtain the vector of the rotated spin axis of the target finger;

[0151] Cross-product the vector of the rotated spin axis of the target finger with the vector of the initial spin axis of the target finger to obtain a cross-product result;

[0152] Calculate the square of the cross product result to obtain the square of the sine value of the target angle; the target angle is the angle between the initial spin axis of the target finger and the spin axis of the target finger after rotation;

[0153] Performing a dot multiplication of the vector of the rotated spin axis of the target finger and the vector of the initial spin axis of the target finger to obtain a cosine value of the target angle;

[0154] A rotation matrix of other rotations of the target finger is obtained according to the square of the sine value of the target angle, the cosine value of the target angle and the cross product result.

[0155] In a possible implementation manner, the neighboring fingers of the target finger include a first neighboring finger and a second neighboring finger, and the fourth determining subunit is specifically configured to:

[0156] Determine the overlapping area of ​​the target finger and the first adjacent finger according to the identification frame position of the target finger and the identification frame position of the first adjacent finger;

[0157] Determine the overlapping area of ​​the target finger and the second adjacent finger according to the identification frame position of the target finger and the identification frame position of the second adjacent finger;

[0158] Calculating the ratio of the overlapping area of ​​the target finger and the first adjacent finger to the area of ​​the target finger to obtain the probability of the target finger and the first adjacent finger overlapping each other; the area of ​​the target finger is determined by the position of the identification frame of the target finger;

[0159] The ratio of the overlapping area of ​​the target finger and the second adjacent finger to the area of ​​the target finger is calculated to obtain the overlapping probability of the target finger and the second adjacent finger.

[0160] In a possible implementation manner, the second determining unit is specifically configured to:

[0161] If the probability of the target finger overlapping with the first adjacent finger or the probability of the target finger overlapping with the second adjacent finger is within a preset probability range, it is determined that the target finger is blocked.

[0162] In a possible implementation manner, the device further includes:

[0163] a second recognition unit, configured to recognize whether a hand region exists in the hand image before the first recognition unit recognizes the posture information of the target finger in the hand image;

[0164] an execution unit, configured to execute the first recognition unit to recognize the posture information of the target finger in the hand image if the hand area exists in the hand image;

[0165] The first generating unit is used to generate a first prompt page if there is no hand area in the hand image, wherein the first prompt page is used to display first prompt information, and the first prompt information is used to prompt that there is no hand area in the hand image.

[0166] In a possible implementation manner, the device further includes:

[0167] The second generating unit is used to generate a second prompt page if it is determined that the target finger is blocked, wherein the second prompt page is used to display second prompt information, and the second prompt information is used to prompt that the target finger is blocked.

[0168] Based on a hand image processing method provided in the above method embodiment, the present application also provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors implement the hand image processing method described in any of the above embodiments.

[0169] Reference below Fig.11, which shows a schematic diagram of the structure of an electronic device 1300 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Fig.11 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0170] like Fig.11 As shown, the electronic device 1300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1306 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 are also stored. The processing device 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0171] Typically, the following devices may be connected to the I / O interface 1305: an input device 1306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1306 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1309. The communication device 1309 may allow the electronic device 1300 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.11 The electronic device 1300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0172] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1309, or installed from the storage device 1306, or installed from the ROM 1302. When the computer program is executed by the processing device 1301, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0173] The electronic device provided in the embodiment of the present application and the hand image processing method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0174] Based on a hand image processing method provided in the above method embodiment, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the hand image processing method as described in any of the above embodiments is implemented.

[0175] It should be noted that the computer-readable medium in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In an embodiment of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0176] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0177] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0178] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the hand image processing method.

[0179] The computer program code for performing the operation of the embodiment of the present application can be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include but are not limited to object-oriented programming languages-such as Java, Smalltalk, C++, and also include conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network-including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0180] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0181] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the unit / module does not, in some cases, constitute a limitation on the unit itself.

[0182] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0183] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0184] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0185] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0186] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0187] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0188] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hand image processing method, characterized in that: The method comprises: Acquire a hand image; Identifying the posture information of the target finger in the hand image; Determining an occlusion degree parameter of the target finger according to the posture information of the target finger; If the occlusion degree parameter of the target finger is within a preset parameter range, it is determined that the target finger is occluded.

2. The method according to claim 1, characterized in that The occlusion degree parameter of the target finger includes a rotation angle of the target finger and / or a probability of overlap between the target finger and an adjacent finger.

3. The method according to claim 2, characterized in that When the occlusion degree of the target finger includes the spin angle of the target finger, determining the occlusion degree parameter of the target finger according to the posture information of the target finger includes: Determining the spin angle of the target finger according to the posture information of the target finger; and determining that the target finger is blocked if the blocking degree parameter of the target finger is within a preset parameter range, including: If the rotation angle of the target finger is within a preset angle range, it is determined that the target finger is blocked.

4. The method according to claim 2, characterized in that: When the occlusion degree of the target finger includes the probability of overlap between the target finger and an adjacent finger, the method further includes: Identifying gesture information of fingers adjacent to the target finger in the hand image; The step of determining the occlusion degree parameter of the target finger according to the posture information of the target finger includes: Determining a recognition frame position of the target finger according to the posture information of the target finger; Determining the position of the recognition frame of the adjacent finger according to the posture information of the adjacent finger; Determining the probability of overlap between the target finger and the adjacent finger based on the position of the recognition frame of the target finger and the position of the recognition frame of the adjacent finger; If the occlusion degree parameter of the target finger is within a preset parameter range, determining that the target finger is occluded includes: If the probability of the target finger overlapping the adjacent finger is within a preset probability range, it is determined that the target finger is blocked.

5. The method according to claim 3, characterized in that: The step of determining the spin angle of the target finger according to the posture information of the target finger includes: According to the posture information of the target finger, obtaining a rotation matrix of other rotations of the target finger; Obtaining a rotation matrix of the rotation axis of the target finger after rotation according to other rotational rotation matrices of the target finger and the rotation matrix in the posture information of the target finger; The spin angle of the target finger is obtained according to the rotation matrix of the spin axis of the target finger after rotation.

6. The method according to claim 5, characterized in that The step of obtaining a rotation matrix of other rotations of the target finger according to the posture information of the target finger includes: Performing matrix multiplication of the rotation matrix in the posture information of the target finger and the vector of the initial spin axis of the target finger to obtain the vector of the rotated spin axis of the target finger; Cross-product the vector of the rotated spin axis of the target finger with the vector of the initial spin axis of the target finger to obtain a cross-product result; Calculate the square of the cross product result to obtain the square of the sine value of the target angle; the target angle is the angle between the initial spin axis of the target finger and the spin axis of the target finger after rotation; Performing a dot multiplication of the vector of the rotated spin axis of the target finger and the vector of the initial spin axis of the target finger to obtain a cosine value of the target angle; A rotation matrix of other rotations of the target finger is obtained according to the square of the sine value of the target angle, the cosine value of the target angle and the cross product result.

7. The method according to claim 4, characterized in that The neighboring fingers of the target finger include a first neighboring finger and a second neighboring finger, and determining the probability of overlap between the target finger and the neighboring finger based on the recognition frame position of the target finger and the recognition frame position of the neighboring finger includes: Determine the overlapping area of ​​the target finger and the first adjacent finger according to the identification frame position of the target finger and the identification frame position of the first adjacent finger; Determine the overlapping area of ​​the target finger and the second adjacent finger according to the identification frame position of the target finger and the identification frame position of the second adjacent finger; Calculating the ratio of the overlapping area of ​​the target finger and the first adjacent finger to the area of ​​the target finger to obtain the probability of the target finger and the first adjacent finger overlapping each other; the area of ​​the target finger is determined by the position of the identification frame of the target finger; The ratio of the overlapping area of ​​the target finger and the second adjacent finger to the area of ​​the target finger is calculated to obtain the overlapping probability of the target finger and the second adjacent finger.

8. The method according to claim 7, characterized in that If the probability of the target finger and the adjacent finger overlapping is within a preset probability range, determining that the target finger is blocked includes: If the probability of the target finger overlapping with the first adjacent finger or the probability of the target finger overlapping with the second adjacent finger is within a preset probability range, it is determined that the target finger is blocked.

9. The method according to any one of claims 1 to 8, characterized in that: Before identifying the posture information of the target finger in the hand image, the method further includes: Identify whether there is a hand area in the hand image; If a hand region exists in the hand image, performing the step of identifying the posture information of the target finger in the hand image; If there is no hand region in the hand image, a first prompt page is generated, wherein the first prompt page is used to display first prompt information, and the first prompt information is used to prompt that there is no hand region in the hand image.

10. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: If it is determined that the target finger is blocked, a second prompt page is generated, where the second prompt page is used to display second prompt information, where the second prompt information is used to prompt that the target finger is blocked.

11. A hand image processing device, characterized in that: The method device: An acquisition unit, used for acquiring a hand image; A recognition unit, used to recognize the posture information of the target finger in the hand image; A first determining unit, configured to determine an occlusion degree parameter of the target finger according to the posture information of the target finger; The second determining unit is configured to determine that the target finger is blocked if the blocking degree parameter of the target finger is within a preset parameter range.

12. A hand image processing device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the hand image processing method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the hand image processing method according to any one of claims 1 to 10.

Images