Cursor mapping method and device, electronic equipment, medium and computer program product

Abstract

The invention provides a cursor mapping method and device, electronic equipment, a medium and a computer program product, and relates to the technical field of machine vision. Shoulder key point coordinates and hand key point coordinates of a target object are obtained in real time; performing distance calculation on the left shoulder key point coordinates and the right shoulder key point coordinates to obtain a double-shoulder distance of the target object, and performing calculation based on the double-shoulder distance, the shoulder key point coordinates and preset virtual frame parameters to obtain a left virtual frame corresponding to the left hand and a right virtual frame corresponding to the right hand; in response to monitoring that the left-hand key point coordinates or the right-hand key point coordinates are located in the corresponding virtual frames, acquiring relative parameters of the coordinates of the hand key points and coordinates of reference angular points on the corresponding virtual frames; and calculating the mapping cursor position of the hand key point coordinate on the projection screen by combining the screen parameter and the coordinate relative parameter of the projection screen. According to the method, the position of the virtual frame is bound to the position of the double-shoulder coordinate position, so that the user can form operation experience by memorizing the action of the user.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a cursor mapping method, device, electronic device, medium and computer program product. Background Art

[0002] With the development of computer technology, in addition to the traditional human-computer interaction method of direct operation through physical devices such as mouse and keyboard, a human-computer interaction method based on air cursor mapping method has also emerged. The air cursor mapping method refers to mapping the cursor position through gestures or other non-contact methods. In the related technology, when users perform air cursor mapping, the accuracy is prone to be low, and as the user moves, the correlation between the user action and the mapping position is not high. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, one purpose of the present application is to propose a cursor mapping method, which obtains the shoulder key point coordinates and hand key point coordinates of the target object in real time, wherein the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates; performs distance calculation on the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the shoulder distance of the target object, and performs calculation based on the shoulder distance, the shoulder key point coordinates and the preset virtual frame parameters to obtain the left virtual frame corresponding to the left hand and the right virtual frame corresponding to the right hand; for any hand key point coordinate in the left hand key point coordinates and the right hand key point coordinates, in response to monitoring that any hand key point coordinate is in its corresponding virtual frame, obtains the coordinate relative parameters of the reference corner points on the hand key point coordinates and its corresponding virtual frame; obtains the screen parameters of the projection screen corresponding to the target object, and calculates the mapping cursor position of the hand key point coordinates on the projection screen in combination with the screen parameters and the coordinate relative parameters.

[0005] The second objective of the present application is to provide a cursor mapping device.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth object of the present application is to provide a non-transitory computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above-mentioned purpose, the first aspect embodiment of the present application proposes a cursor mapping method, including: acquiring the shoulder key point coordinates and hand key point coordinates of the target object in real time, wherein the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates; performing distance calculation on the left shoulder key point coordinates and the right shoulder key point coordinates to acquire the shoulder distance of the target object, and performing calculation based on the shoulder distance, the shoulder key point coordinates and the preset virtual frame parameters to acquire the left virtual frame corresponding to the left hand and the right virtual frame corresponding to the right hand; for any hand key point coordinate in the left hand key point coordinates and the right hand key point coordinates, in response to monitoring that any hand key point coordinate is in its corresponding virtual frame, acquiring the coordinate relative parameters of the reference corner points on the hand key point coordinates and its corresponding virtual frame; acquiring the screen parameters of the projection screen corresponding to the target object, and calculating the mapping cursor position of the hand key point coordinates on the projection screen in combination with the screen parameters and the coordinate relative parameters.

[0010] According to one embodiment of the present application, the shoulder key point coordinates and hand key point coordinates of the target object are acquired in real time, including: acquiring a real-time video stream of the target object, and extracting a target image corresponding to the target object based on the real-time video stream; inputting the target image into a shoulder key point recognition network to acquire the shoulder key point coordinates output by the shoulder key point recognition network; inputting the target image into a hand key point recognition network to acquire the hand key point coordinates output by the hand key point recognition network.

[0011] According to an embodiment of the present application, a calculation is performed based on the shoulder distance, the coordinates of the shoulder key points and the preset virtual frame parameters to obtain a left virtual frame corresponding to the left hand and a right virtual frame corresponding to the right hand, including: presetting virtual frame parameters corresponding to the virtual frames, wherein the virtual frame parameters at least include a length parameter, a height parameter, a horizontal axis offset parameter and a vertical axis offset parameter of the virtual frame; taking the product of the shoulder distance and the length parameter as the virtual frame length of the virtual frame, and taking the product of the shoulder distance and the height parameter as the virtual frame height of the virtual frame; taking the product of the horizontal axis offset parameter and the virtual frame length as the virtual frame height; taking the product of the horizontal axis offset parameter and the virtual frame length as the virtual frame height; taking the product of the horizontal axis offset parameter and the virtual frame length as the virtual frame height; taking the product of the horizontal axis offset parameter and the virtual frame length as the virtual frame height. Half of the product of the vertical axis offset parameter and the height of the virtual frame is used as the horizontal axis offset distance corresponding to the reference corner point of the virtual frame, and half of the product of the vertical axis offset parameter and the height of the virtual frame is used as the vertical axis offset distance corresponding to the reference corner point of the virtual frame; for any shoulder key point coordinates of the left shoulder key point coordinates and the right shoulder key point coordinates, the shoulder key point coordinates, the horizontal axis offset distance and the vertical axis offset distance are calculated to obtain the reference corner point coordinates of the reference corner point of the virtual frame corresponding to the shoulder key point coordinates; according to the reference corner point coordinates, the virtual frame length and the virtual frame height are combined to expand the virtual frame to generate the left virtual frame and the right virtual frame.

[0012] According to one embodiment of the present application, the shoulder key point coordinates, the horizontal axis offset distance and the vertical axis offset distance are calculated to obtain the reference corner point coordinates of the reference corner point, including: in response to the reference corner point being the upper left starting corner point of the virtual frame, the sum of the horizontal coordinate of the shoulder key point coordinates and the horizontal axis offset distance is used as the horizontal coordinate of the reference corner point, and the sum of the vertical coordinate of the shoulder key point coordinates and the vertical axis offset distance is used as the vertical coordinate of the reference corner point; the reference corner point coordinates are determined according to the horizontal coordinate and the vertical coordinate of the reference corner point.

[0013] According to one embodiment of the present application, the mapping cursor position of the hand key point coordinates on the projection screen is calculated in combination with the screen parameters and the coordinate relative parameters, including: obtaining the number of horizontal pixels and the number of vertical pixels of the projection screen; obtaining a first ratio of the horizontal coordinate value in the coordinate relative parameters to the length of the virtual frame, and a second ratio of the vertical coordinate value in the coordinate relative parameters to the height of the virtual frame; determining the target column from the number of horizontal pixels according to the first ratio, and determining the target row from the number of vertical pixels according to the second ratio, and using the pixel points corresponding to the target row and the target column as the mapping cursor position.

[0014] According to an embodiment of the present application, the cursor mapping method further includes: updating the left virtual frame and the right virtual frame in real time.

[0015] To achieve the above-mentioned purpose, the second aspect of the present application proposes a cursor mapping device, including: an acquisition module, used to acquire the shoulder key point coordinates and hand key point coordinates of the target object in real time, wherein the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates; a calculation module, used to perform distance calculation on the left shoulder key point coordinates and the right shoulder key point coordinates, acquire the double shoulder distance of the target object, and perform calculation based on the double shoulder distance, the shoulder key point coordinates and the preset virtual frame parameters to acquire the left virtual frame corresponding to the left hand and the right virtual frame corresponding to the right hand; a monitoring module, used to obtain the coordinate relative parameters of the hand key point coordinates and the reference corner points on the corresponding virtual frame in response to monitoring that any hand key point coordinate is in its corresponding virtual frame for any hand key point coordinate among the left hand key point coordinates and the right hand key point coordinates; a projection module, used to acquire the screen parameters of the projection screen corresponding to the target object, and calculate the mapping cursor position of the hand key point coordinates on the projection screen in combination with the screen parameters and the coordinate relative parameters.

[0016] According to one embodiment of the present application, the acquisition module is also used to: collect the real-time video stream of the target object, and extract the target image corresponding to the target object based on the real-time video stream; input the target image into the shoulder key point recognition network to obtain the shoulder key point coordinates output by the shoulder key point recognition network; input the target image into the hand key point recognition network to obtain the hand key point coordinates output by the hand key point recognition network.

[0017] According to one embodiment of the present application, the calculation module is further used to: pre-set virtual frame parameters corresponding to the virtual frame, wherein the virtual frame parameters include at least a length parameter, a height parameter, a horizontal axis offset parameter and a vertical axis offset parameter of the virtual frame; use the product of the double-shoulder distance and the length parameter as the virtual frame length of the virtual frame, and use the product of the double-shoulder distance and the height parameter as the virtual frame height of the virtual frame; use half of the product of the horizontal axis offset parameter and the virtual frame length as the horizontal axis offset distance corresponding to the reference corner point of the virtual frame, and use half of the product of the vertical axis offset parameter and the virtual frame height as the vertical axis offset distance corresponding to the reference corner point of the virtual frame; for any shoulder key point coordinates in the left shoulder key point coordinates and the right shoulder key point coordinates, calculate the shoulder key point coordinates, the horizontal axis offset distance and the vertical axis offset distance to obtain the reference corner point coordinates of the reference corner point of the virtual frame corresponding to the shoulder key point coordinates; and expand the virtual frame according to the reference corner point coordinates in combination with the virtual frame length and the virtual frame height to generate a left virtual frame and a right virtual frame.

[0018] According to one embodiment of the present application, the calculation module is also used for: in response to the reference corner point being the upper left starting corner point of the virtual frame, taking the sum of the horizontal coordinate of the shoulder key point coordinate and the horizontal axis offset distance as the horizontal coordinate of the reference corner point, and taking the sum of the vertical coordinate of the shoulder key point coordinate and the vertical axis offset distance as the vertical coordinate of the reference corner point; determining the reference corner point coordinates according to the horizontal coordinate and the vertical coordinate of the reference corner point.

[0019] According to one embodiment of the present application, the projection module is also used to: obtain the number of horizontal pixels and the number of vertical pixels of the projection screen; obtain a first ratio of the horizontal coordinate value in the coordinate relative parameter to the length of the virtual frame, and a second ratio of the vertical coordinate value in the coordinate relative parameter to the height of the virtual frame; determine the target column from the number of horizontal pixels according to the first ratio, and determine the target row from the number of vertical pixels according to the second ratio, and use the pixel points corresponding to the target row and the target column as the mapping cursor position.

[0020] According to an embodiment of the present application, the computing module is further used to: update the left virtual frame and the right virtual frame in real time.

[0021] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the cursor mapping method as described in the first aspect embodiment of the present application.

[0022] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the cursor mapping method as described in the first aspect embodiment of the present application.

[0023] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the cursor mapping method as described in the first aspect of the present application.

[0024] The present application achieves at least the following beneficial effects: the cursor mapping method proposed in the present application has high mapping accuracy, and in the present application the distance between the human shoulders and the size of the virtual frame are bound, ensuring that different users can obtain a good operating experience when they are at different distances from the camera. At the same time, the position of the virtual frame is bound by the coordinate position of the shoulders, so that the user can form operating experience by remembering his own actions when using gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 It is a schematic diagram of an exemplary implementation of a cursor mapping method shown in an embodiment of the present application.

[0027] Figure 2 It is a schematic diagram of key points of a hand shown in an embodiment of the present application.

[0028] Figure 3 It is a schematic diagram of a key point of the shoulder shown in an embodiment of the present application.

[0029] Figure 4 It is a schematic diagram of a left virtual frame corresponding to the left hand and a right virtual frame corresponding to the right hand shown in an embodiment of the present application.

[0030] Figure 5 This is a schematic diagram showing a method of determining a position of a mapping cursor on a projection screen according to an embodiment of the present application.

[0031] Figure 6 It is a schematic diagram of an exemplary implementation of a cursor mapping method shown in an embodiment of the present application.

[0032] Figure 7 It is a schematic diagram of a cursor mapping device shown in an embodiment of the present application.

[0033] Figure 8 It is a schematic diagram of an electronic device shown in one embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0036] Figure 1 is a schematic diagram of an exemplary implementation of a cursor mapping method shown in the present application, such as Figure 1 As shown, the cursor mapping method comprises the following steps:

[0037] S101, acquiring the shoulder key point coordinates and the hand key point coordinates of the target object in real time, wherein the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates.

[0038] The user who is performing air cursor mapping is taken as the target object. During the air cursor mapping process of the target object, the video stream of the target object is collected in real time based on the image acquisition device, and the video stream is processed to obtain the shoulder key point coordinates and the hand key point coordinates of the target object in real time. The image acquisition device can be set at the projection screen, and the target object faces the projection screen to perform gesture interaction.

[0039] In the present application, the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates.

[0040] Figure 2 is a schematic diagram of a hand key point shown in this application, such as Figure 2 As shown, the left or right hand of the target object contains multiple candidate key points ( Figure 2 In this application, for the convenience of calculation, the coordinates of one of the candidate key points of the left hand are used as the left hand key point coordinates corresponding to the left hand in this application (when Figure 2 When the left hand is represented, the coordinates of the black circle are the left hand key point coordinates), and the coordinates of one of the multiple candidate key points of the right hand are used as the right hand key point coordinates corresponding to the right hand in this application (when Figure 2 When it represents the right hand, the coordinates of the black circle are the coordinates of the key points of the right hand).

[0041] S102, calculate the distance between the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the shoulder distance of the target object, and calculate based on the shoulder distance, the shoulder key point coordinates and the preset virtual frame parameters to obtain the left virtual frame corresponding to the left hand and the right virtual frame corresponding to the right hand.

[0042] Figure 3 is a schematic diagram of a shoulder key point shown in the present application, such as Figure 3 As shown, assuming Figure 3 The target object in the image is standing facing forward. Figure 3 The two black circles in the figure respectively show the left shoulder key point and the right shoulder key point of the target object. In this application, the coordinates of the left shoulder key point are obtained as the left shoulder key point coordinates, and the left shoulder key point coordinates are marked as S1(x s1 ,y s1 , z s1 ), obtain the coordinates of the right shoulder key point as the right shoulder key point coordinates, and mark the right shoulder key point coordinates as S2(x s2 ,y s2 , z s2 ). The Z axis represents the coordinate depth between the left shoulder or the right shoulder and the image acquisition device. The smaller the Z value, the closer the coordinate is to the image acquisition device.

[0043] Calculate the distance between the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the double shoulder distance of the target object, and record the double shoulder distance as D. Then

[0044] Preset virtual frame parameters, determine the left virtual frame corresponding to the left hand according to the left shoulder key point coordinates, the shoulder distance D and the virtual frame parameters; and determine the right virtual frame corresponding to the right hand according to the right shoulder key point coordinates, the shoulder distance D and the virtual frame parameters. Exemplarily, the virtual frame parameters may include a length parameter p and a height parameter q, the length parameter p is used to represent the ratio of the length of the virtual frame to the shoulder distance D, and the height parameter q is used to represent the ratio of the height of the virtual frame to the shoulder distance D. Taking the determination of the left virtual frame as an example, the product of the length parameter p and the shoulder distance D can be used as the length value W of the left virtual frame, and the product of the height parameter q and the shoulder distance D can be used as the height value H of the left virtual frame. Based on the left shoulder key point coordinates, the left virtual frame is expanded according to the length value and the height value of the left virtual frame to obtain the final determined left virtual frame. Among them, the Z axis of the virtual frame, that is, the depth of the virtual frame, is based on the depth value of the hand.

[0045] Figure 4 is a schematic diagram of a left virtual frame corresponding to the left hand and a right virtual frame corresponding to the right hand shown in the present application, such as Figure 4As shown, the left shoulder of the target object corresponds to the left virtual frame, and accordingly, the left virtual frame can be understood as the operating area of ​​the left hand, that is, it can also be understood that the left hand corresponds to the left virtual frame; the right shoulder of the target object corresponds to the right virtual frame, and accordingly, the right virtual frame can be understood as the operating area of ​​the right hand, that is, it can also be understood that the right hand corresponds to the right virtual frame.

[0046] S103, for any hand key point coordinate among the left hand key point coordinates and the right hand key point coordinates, in response to monitoring that any hand key point coordinate is in its corresponding virtual frame, obtain the coordinate relative parameters of the hand key point coordinate and the reference corner point on the corresponding virtual frame.

[0047] The upper left corner points of the left virtual frame and the right virtual frame are respectively used as the reference corner points corresponding to the virtual frames.

[0048] Get the coordinates of the reference corner of the left virtual frame and label it as P1(x p1 ,y p1 , z p1 ).

[0049] Get the coordinates of the reference corner of the right virtual frame and label it as P2(x p2 ,y p2 , z p2 ).

[0050] Mark the left hand key point coordinate as F1(x f1 ,y f1 , z f1 ), if the left-hand key point coordinates are detected to be in the left virtual frame, the coordinate relative parameters of the left-hand key point coordinates F1 and the corresponding reference corner point P1 on the left virtual frame are obtained, and the coordinate relative parameters are recorded as O1(xo1, yo1, zo1), where xo1 = x f1 -x p1 ; yo1 = y f1 -y p1 ;zo1=z f1 -z p1 .

[0051] Mark the right hand key point coordinate as F2(x f2 ,y f2 , z f2 ), if the coordinates of the right-hand key point are detected to be in the right virtual frame, the coordinate relative parameters of the right-hand key point coordinate F2 and the reference corner point P2 on the corresponding right virtual frame are obtained, and the coordinate relative parameters are recorded as O2(xo2, yo2, zo2), where xo2 = x f2 -x p2 ; yo2 = y f2 -y p2 ;zo2=z f2 -zp2 .

[0052] S104, obtaining screen parameters of the projection screen corresponding to the target object, and calculating the mapping cursor position of the hand key point coordinates on the projection screen in combination with the screen parameters and the coordinate relative parameters.

[0053] Get the screen parameters of the projection screen corresponding to the target object, which include the number of horizontal pixels S of the projection screen. W And the number of vertical pixels S h .

[0054] Figure 5 is a schematic diagram of determining the position of a mapping cursor on a projection screen shown in the present application, such as Figure 5 As shown, if the right hand key point coordinates are detected to be in the right virtual frame, the coordinate relative parameters O2 (xo2, yo2, zo2) of the right hand key point coordinates F2 and the corresponding reference corner point P2 on the right virtual frame and the screen parameters are combined to determine the mapping cursor position. In this application, the cursor mapping is only related to the X-axis coordinates and the Y-axis coordinates. The mapping position of the right hand key point coordinates F2 on the projection screen is S2'(x sr ,y sr ),in,

[0055] Similar to the situation when the right-hand key point coordinates are in the right virtual frame, if the left-hand key point coordinates are detected to be in the left virtual frame, the mapping cursor position is determined by combining the above-obtained left-hand key point coordinates F1 with the corresponding reference corner point P1 on the left virtual frame and the screen parameters.

[0056] In this application, cursor mapping is only related to the X-axis coordinate and the Y-axis coordinate, so the mapping position of the left-hand key point coordinate F1 on the projection screen is S1'(x sl ,y sl ),in,

[0057] The embodiment of the present application proposes a cursor mapping method, which obtains the shoulder key point coordinates and hand key point coordinates of the target object in real time, wherein the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates; performs distance calculation on the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the double-shoulder distance of the target object, and performs calculation based on the double-shoulder distance, the shoulder key point coordinates and the preset virtual frame parameters to obtain the left virtual frame corresponding to the left hand and the right virtual frame corresponding to the right hand; for any hand key point coordinate among the left hand key point coordinates and the right hand key point coordinates, in response to monitoring that any hand key point coordinate is in its corresponding virtual frame, obtains the coordinate relative parameters of the reference corner points on the hand key point coordinates and its corresponding virtual frame; obtains the screen parameters of the projection screen corresponding to the target object, and calculates the mapping cursor position of the hand key point coordinates on the projection screen in combination with the screen parameters and the coordinate relative parameters. The cursor mapping method proposed in the present application has high mapping accuracy, and in the present application, the distance between the human shoulders and the size of the virtual frame are bound, ensuring that different users can get a good operating experience when they are at different distances from the camera. At the same time, the position of the virtual frame is bound by the coordinate position of the shoulders, so that the user can form operating experience by remembering his own actions when using gestures.

[0058] Furthermore, since the position of the target object may be constantly moving, in this application, in order to ensure the accuracy of cursor mapping, the left virtual frame and the right virtual frame need to be updated in real time.

[0059] Figure 6 is a schematic diagram of an exemplary implementation of a cursor mapping method shown in the present application, such as Figure 6 As shown, the cursor mapping method comprises the following steps:

[0060] S601, collecting a real-time video stream of a target object, and extracting a target image corresponding to the target object based on the real-time video stream.

[0061] The image acquisition device acquires the video stream of the target object in real time, and performs image extraction on the video stream to extract the target image corresponding to the target object.

[0062] S602, inputting the target image into a shoulder key point recognition network to obtain shoulder key point coordinates output by the shoulder key point recognition network, where the shoulder key point coordinates include left shoulder key point coordinates and right shoulder key point coordinates.

[0063] Input the target image into the trained shoulder key point recognition network to obtain the shoulder key point coordinates output by the shoulder key point recognition network. The shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates. The left shoulder key point coordinates are marked as S1(x s1 ,ys1 , z s1 ), and mark the coordinates of the right shoulder key point as S2(x s2 ,y s2 , z s2 ).

[0064] Optionally, the shoulder key point recognition network can adopt the MediaPipe Pose model under the MediaPipe framework.

[0065] Optionally, image sampling may be performed on the sampled object and shoulder key points may be annotated on the sampled image to iteratively train the neural network model and obtain a shoulder key point recognition network generated after training.

[0066] S603, inputting the target image into the hand key point recognition network to obtain the hand key point coordinates output by the hand key point recognition network, where the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates.

[0067] Input the target image into the trained hand key point recognition network to obtain the hand key point coordinates output by the hand key point recognition network. The hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates. The left hand key point coordinates are marked as F1(x f1 ,y f1 , z f1 ), mark the coordinates of the right-hand key point as F2(x f2 ,y f2 , z f2 ).

[0068] Optionally, the hand key point recognition network can adopt the MediaPipe Hands model under the MediaPipe architecture.

[0069] Optionally, image sampling may be performed on the sampled object and hand key points may be annotated on the sampled image to iteratively train the neural network model and obtain a hand key point recognition network generated after training.

[0070] S604, calculate the distance between the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the shoulder distance of the target object, and calculate based on the shoulder distance, the shoulder key point coordinates and the preset virtual frame parameters to obtain the left virtual frame corresponding to the left hand and the right virtual frame corresponding to the right hand.

[0071] Calculate the distance between the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the double shoulder distance of the target object, and record the double shoulder distance as D. Then

[0072] Preset virtual frame parameters corresponding to the virtual frame, wherein the virtual frame parameters at least include a length parameter p, a height parameter q, and a horizontal axis offset parameter x of the virtual frame. of and the longitudinal axis offset parameter y of Among them, due to the different operating habits of each user, some users are more accustomed to operating in the right area of ​​the shoulder, and some users are more accustomed to operating in the left area of ​​the shoulder. In order to improve the user's comfort, the horizontal axis offset parameter x is introduced in this application. of and the longitudinal axis offset parameter y of , used to limit the offset of the virtual frame.

[0073] The product of the shoulder distance D and the length parameter p is used as the virtual frame length of the virtual frame, and the virtual frame length is recorded as W, then W=p*D.

[0074] The product of the shoulder distance D and the height parameter q is used as the virtual frame height of the virtual frame, and the virtual frame height is recorded as H, then H=q*D.

[0075] Offset the horizontal axis by parameter x of Half of the product of the virtual frame length is used as the horizontal axis offset distance corresponding to the reference corner point of the virtual frame.

[0076] Offset the vertical axis by parameter y of Half of the product of the virtual frame height is used as the vertical axis offset distance corresponding to the reference corner point of the virtual frame.

[0077] For any shoulder key point coordinates among the left shoulder key point coordinates and the right shoulder key point coordinates, the shoulder key point coordinates, the horizontal axis offset distance and the vertical axis offset distance are calculated to obtain the reference corner point coordinates of the reference corner point of the virtual frame corresponding to the shoulder key point coordinates.

[0078] If the reference corner point is limited to the starting corner point of the upper left corner of the virtual frame, the sum of the horizontal coordinate of the shoulder key point coordinate and the horizontal axis offset distance is used as the horizontal coordinate of the reference corner point, and the sum of the vertical coordinate of the shoulder key point coordinate and the vertical axis offset distance is used as the vertical coordinate of the reference corner point. The coordinates of the reference corner point are determined according to the horizontal coordinate and the vertical coordinate of the reference corner point, and the coordinates of the reference corner point of the left virtual frame are marked as P1(x p1 ,y p1 , z p1 ),in, z p1 =z f1 ; Mark the coordinates of the reference corner of the right virtual frame as P2(x p2 ,y p2 , z p2 ),in, zp1 =z f2 .

[0079] The virtual frame is extended according to the reference corner coordinates in combination with the virtual frame length and the virtual frame height to generate a left virtual frame and a right virtual frame.

[0080] S605, for any hand key point coordinate among the left hand key point coordinates and the right hand key point coordinates, in response to monitoring that any hand key point coordinate is in its corresponding virtual frame, obtain the coordinate relative parameters of the hand key point coordinate and the reference corner point on the corresponding virtual frame.

[0081] If the coordinates of the left-hand key point are detected to be in the left virtual frame, the coordinate relative parameters of the left-hand key point coordinate F1 and the reference corner point P1 on the left virtual frame corresponding to it are obtained, and the coordinate relative parameters are recorded as O1(xo1, yo1, zo1), where xo1 = x f1 -x p1 ; yo1 = y f1 -y p1 ;zo1=z f1 -z p1 .

[0082] If the coordinates of the right-hand key point are detected to be in the right virtual frame, the coordinate relative parameters of the right-hand key point coordinate F2 and the reference corner point P2 on the corresponding right virtual frame are obtained, and the coordinate relative parameters are recorded as O2(xo2, yo2, zo2), where xo2 = x f2 -x p2 ; yo2 = y f2 -y p2 ;zo2=z f2 -z p2 .

[0083] S606, obtaining the number of horizontal pixels and the number of vertical pixels of the projection screen.

[0084] Get the screen parameters of the projection screen corresponding to the target object, which include the number of horizontal pixels S of the projection screen. W And the number of vertical pixels S h .

[0085] S607, obtaining a first ratio of the horizontal coordinate value in the coordinate relative parameter to the length of the virtual frame corresponding to the virtual frame, and a second ratio of the vertical coordinate value in the coordinate relative parameter to the height of the virtual frame corresponding to the virtual frame.

[0086] Taking the coordinates of the left-hand key point in the left virtual frame as an example, determine the horizontal coordinate value of the coordinate relative parameter of the left-hand key point coordinate and the reference corner point on the left virtual frame corresponding to it, and take the ratio of the horizontal coordinate value to the length of the virtual frame corresponding to the left virtual frame as the first ratio, that is, the first ratio can be recorded as Determine the ordinate value of the coordinate relative parameter of the left-hand key point coordinate and the reference corner point on the left virtual frame corresponding to it, and take the ratio of the ordinate value to the height of the left virtual frame corresponding to the virtual frame as the second ratio, that is, the second ratio can be recorded as

[0087] S608, determining a target column from the number of horizontal pixels according to a first ratio, and determining a target row from the number of vertical pixels according to a second ratio, and using the pixels corresponding to the target row and the target column as the mapping cursor position.

[0088] The target column is determined from the number of horizontal pixels according to the first ratio, and the target row is determined from the number of vertical pixels according to the second ratio, and the pixels corresponding to the target row and the target column are used as the mapping cursor position. Taking the left hand key point coordinates in the left virtual frame as an example, the mapping position of the left hand key point coordinates F1 on the projection screen is S1'(x sl ,y sl ),in,

[0089] The cursor mapping method proposed in the present application has high mapping accuracy, and in the present application, the distance between the human shoulders and the size of the virtual frame are bound, ensuring that different users can get a good operating experience when they are at different distances from the camera. At the same time, the position of the virtual frame is bound by the coordinate position of the shoulders, so that the user can form operating experience by remembering his own actions when using gestures.

[0090] Figure 7 is a schematic diagram of a cursor mapping device shown in the present application, such as Figure 7 As shown, the cursor mapping device 700 includes an acquisition module 701, a calculation module 702, a monitoring module 703 and a projection module 704, wherein:

[0091] The acquisition module 701 is used to acquire the shoulder key point coordinates and hand key point coordinates of the target object in real time, wherein the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates.

[0092] The calculation module 702 is used to calculate the distance between the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the shoulder distance of the target object, and to calculate based on the shoulder distance, the shoulder key point coordinates and the preset virtual frame parameters to obtain the left virtual frame corresponding to the left hand and the right virtual frame corresponding to the right hand.

[0093] The monitoring module 703 is used to obtain the coordinate relative parameters of the hand key point coordinate and the reference corner point on the corresponding virtual frame in response to monitoring that any hand key point coordinate is in its corresponding virtual frame.

[0094] The projection module 704 is used to obtain the screen parameters of the projection screen corresponding to the target object, and calculate the mapping cursor position of the hand key point coordinates on the projection screen in combination with the screen parameters and the coordinate relative parameters.

[0095] In this device, the distance between the human shoulders and the size of the virtual frame are bound to ensure that different users can get a good operating experience when they are at different distances from the camera. At the same time, the position of the virtual frame is bound using the coordinate position of the shoulders, so that users can form operating experience through the memory of their own actions when using gestures.

[0096] According to one embodiment of the present application, the acquisition module 701 is also used to: collect the real-time video stream of the target object, and extract the target image corresponding to the target object based on the real-time video stream; input the target image into the shoulder key point recognition network to obtain the shoulder key point coordinates output by the shoulder key point recognition network; input the target image into the hand key point recognition network to obtain the hand key point coordinates output by the hand key point recognition network.

[0097] According to one embodiment of the present application, the calculation module 702 is further used to: pre-set virtual frame parameters corresponding to the virtual frame, wherein the virtual frame parameters include at least a length parameter, a height parameter, a horizontal axis offset parameter and a vertical axis offset parameter of the virtual frame; use the product of the double-shoulder distance and the length parameter as the virtual frame length of the virtual frame, and use the product of the double-shoulder distance and the height parameter as the virtual frame height of the virtual frame; use half of the product of the horizontal axis offset parameter and the virtual frame length as the horizontal axis offset distance corresponding to the reference corner point of the virtual frame, and use half of the product of the vertical axis offset parameter and the virtual frame height as the vertical axis offset distance corresponding to the reference corner point of the virtual frame; for any shoulder key point coordinates in the left shoulder key point coordinates and the right shoulder key point coordinates, calculate the shoulder key point coordinates, the horizontal axis offset distance and the vertical axis offset distance to obtain the reference corner point coordinates of the reference corner point of the virtual frame corresponding to the shoulder key point coordinates; and expand the virtual frame according to the reference corner point coordinates in combination with the virtual frame length and the virtual frame height to generate a left virtual frame and a right virtual frame.

[0098] According to one embodiment of the present application, the calculation module 702 is also used for: in response to the reference corner point being the upper left starting corner point of the virtual frame, taking the sum of the horizontal coordinate of the shoulder key point coordinate and the horizontal axis offset distance as the horizontal coordinate of the reference corner point, and taking the sum of the vertical coordinate of the shoulder key point coordinate and the vertical axis offset distance as the vertical coordinate of the reference corner point; determining the reference corner point coordinates according to the horizontal coordinate and the vertical coordinate of the reference corner point.

[0099] According to one embodiment of the present application, the projection module 704 is also used to: obtain the number of horizontal pixels and the number of vertical pixels of the projection screen; obtain a first ratio of the horizontal coordinate value in the coordinate relative parameter to the length of the virtual frame, and a second ratio of the vertical coordinate value in the coordinate relative parameter to the height of the virtual frame; determine the target column from the number of horizontal pixels according to the first ratio, and determine the target row from the number of vertical pixels according to the second ratio, and use the pixel points corresponding to the target row and the target column as the mapping cursor position.

[0100] According to an embodiment of the present application, the calculation module 702 is further used to: update the left virtual frame and the right virtual frame in real time.

[0101] In order to implement the above embodiment, the present application embodiment also proposes an electronic device 800, such as Figure 8 As shown, the electronic device 800 includes: a processor 801 and a memory 802 communicatively connected to the processor, the memory 802 stores instructions executable by at least one processor, and the instructions are executed by at least one processor 801 to implement the cursor mapping method shown in the above embodiment.

[0102] In order to implement the above embodiment, the embodiment of the present application further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to implement the cursor mapping method shown in the above embodiment.

[0103] In order to implement the above embodiment, the embodiment of the present application also proposes a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the cursor mapping method shown in the above embodiment.

[0104] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0106] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0107] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A cursor mapping method, characterized in that: include: Acquire the shoulder key point coordinates and the hand key point coordinates of the target object in real time, wherein the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates; Performing distance calculation on the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the double-shoulder distance of the target object, and performing calculation based on the double-shoulder distance, the shoulder key point coordinates and preset virtual frame parameters to obtain a left virtual frame corresponding to the left hand and a right virtual frame corresponding to the right hand; For any hand key point coordinate among the left hand key point coordinate and the right hand key point coordinate, in response to monitoring that any hand key point coordinate is in its corresponding virtual frame, obtaining coordinate relative parameters of the hand key point coordinate and a reference corner point on the corresponding virtual frame; The screen parameters of the projection screen corresponding to the target object are obtained, and the mapping cursor position of the hand key point coordinates on the projection screen is calculated by combining the screen parameters and the coordinate relative parameters.

2. The cursor mapping method according to claim 1, characterized in that: The real-time acquisition of the shoulder key point coordinates and the hand key point coordinates of the target object includes: Collecting a real-time video stream of the target object, and extracting a target image corresponding to the target object based on the real-time video stream; Inputting the target image into a shoulder key point recognition network to obtain the shoulder key point coordinates output by the shoulder key point recognition network; The target image is input into a hand key point recognition network to obtain the hand key point coordinates output by the hand key point recognition network.

3. The cursor mapping method according to claim 1, characterized in that: The step of calculating based on the distance between the two shoulders, the coordinates of the shoulder key points and preset virtual frame parameters to obtain a left virtual frame corresponding to the left hand and a right virtual frame corresponding to the right hand includes: Presetting virtual frame parameters corresponding to the virtual frame, wherein the virtual frame parameters at least include a length parameter, a height parameter, a horizontal axis offset parameter, and a vertical axis offset parameter of the virtual frame; The product of the double-shoulder distance and the length parameter is used as the virtual frame length of the virtual frame, and the product of the double-shoulder distance and the height parameter is used as the virtual frame height of the virtual frame; Using half of the product of the horizontal axis offset parameter and the length of the virtual frame as the horizontal axis offset distance corresponding to the reference corner point of the virtual frame, and using half of the product of the vertical axis offset parameter and the height of the virtual frame as the vertical axis offset distance corresponding to the reference corner point of the virtual frame; For any one of the shoulder key point coordinates of the left shoulder key point coordinates and the right shoulder key point coordinates, the shoulder key point coordinates, the horizontal axis offset distance, and the vertical axis offset distance are calculated to obtain reference corner point coordinates of a reference corner point of a virtual frame corresponding to the shoulder key point coordinates; The virtual frame is extended according to the reference corner point coordinates in combination with the virtual frame length and the virtual frame height to generate the left virtual frame and the right virtual frame.

4. The cursor mapping method according to claim 3, characterized in that: The step of calculating the shoulder key point coordinates, the horizontal axis offset distance, and the vertical axis offset distance to obtain the reference corner point coordinates of the reference corner point includes: In response to the reference corner point being the upper left starting corner point of the virtual frame, the sum of the abscissa of the shoulder key point coordinates and the abscissa offset distance is used as the abscissa of the reference corner point, and the sum of the ordinate of the shoulder key point coordinates and the abscissa offset distance is used as the ordinate of the reference corner point; The coordinates of the reference corner point are determined according to the abscissa of the reference corner point and the ordinate of the reference corner point.

5. The cursor mapping method according to claim 4, characterized in that: The step of calculating the mapping cursor position of the hand key point coordinates on the projection screen by combining the screen parameters and the coordinate relative parameters includes: Obtaining the number of horizontal pixels and the number of vertical pixels of the projection screen; Acquire a first ratio of the horizontal coordinate value in the coordinate relative parameter to the length of the virtual frame, and a second ratio of the vertical coordinate value in the coordinate relative parameter to the height of the virtual frame; The target column is determined from the number of horizontal pixels according to a first ratio, and the target row is determined from the number of vertical pixels according to a second ratio, and the pixel points corresponding to the target row and the target column are used as the mapping cursor position.

6. The cursor mapping method according to any one of claims 1 to 5, characterized in that: The method further comprises: The left virtual frame and the right virtual frame are updated in real time.

7. A cursor mapping device, characterized in that: include: An acquisition module, used for acquiring the shoulder key point coordinates and the hand key point coordinates of the target object in real time, wherein the shoulder key point coordinates include the left shoulder key point coordinates and the right shoulder key point coordinates, and the hand key point coordinates include the left hand key point coordinates and the right hand key point coordinates; a calculation module, configured to perform distance calculation on the left shoulder key point coordinates and the right shoulder key point coordinates to obtain the double-shoulder distance of the target object, and perform calculation based on the double-shoulder distance, the shoulder key point coordinates and preset virtual frame parameters to obtain a left virtual frame corresponding to the left hand and a right virtual frame corresponding to the right hand; A monitoring module, for obtaining, for any hand key point coordinate among the left hand key point coordinate and the right hand key point coordinate, coordinate relative parameters of a reference corner point on the virtual frame corresponding to the hand key point coordinate in response to monitoring that any hand key point coordinate is in the virtual frame corresponding to the hand key point coordinate; The projection module is used to obtain the screen parameters of the projection screen corresponding to the target object, and calculate the mapping cursor position of the hand key point coordinates on the projection screen in combination with the screen parameters and the coordinate relative parameters.

8. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps according to any one of claims 1 to 6.

Images