Image key point labeling method and device, electronic equipment and storage medium

Abstract

The invention provides an image key point labeling method and device, electronic equipment and a computer readable storage medium. The image key point labeling method comprises the steps that gesture key point labeling information of an initial hand frame is acquired, the initial hand frame is an image containing a calibration plate, and a plurality of reference points are arranged on the calibration plate; obtaining a relative pose between a camera and a calibration plate when the initial hand frame is collected, and taking the relative pose as a first camera pose; acquiring a relative pose between the camera and the calibration plate when the current hand frame is acquired, and taking the relative pose as a second camera pose; and based on the first camera pose, the second camera pose and the gesture key point labeling information of the initial hand frame, performing key point labeling on the current hand frame to obtain the gesture key point labeling information of the current hand frame. According to the method, the image labeling efficiency and the image labeling precision can be improved under the condition that the fitting degree of the labeled image and the real image is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image key point labeling method, device, electronic device and computer-readable storage medium. Background Art

[0002] In some specific scenarios, hand movements are often tracked by annotating gesture key points to determine whether the hand movements meet the requirements. At present, the annotation methods of gesture key points can be roughly divided into two categories: real hand annotation method and virtual hand digital modeling method. The real hand annotation method is to first make the required gesture, and then use the camera to collect hand images or videos for manual annotation. The annotated image has a high degree of fit with the real image, but the image annotation efficiency is relatively low. Among them, the virtual hand digital modeling method arbitrarily replaces the background, angle and lighting conditions of the hand, and the image annotation efficiency is relatively high, but the annotated image obtained is different from the real image. Therefore, how to improve the image annotation efficiency while ensuring the fit between the annotated image and the real image has become an important research issue in image annotation. Summary of the invention

[0003] The present application provides a method, device, electronic device and computer-readable storage medium for labeling key points of an image, which can improve the efficiency and accuracy of image labeling while ensuring the consistency between the labeled image and the real image.

[0004] In a first aspect, the present application provides a method for labeling key points of an image, the method comprising:

[0005] Acquire gesture key point annotation information of an initial hand frame, wherein the initial hand frame is an image including a calibration plate, and a plurality of reference points are set on the calibration plate;

[0006] Obtaining the relative posture between the camera and the calibration plate when the initial hand frame is collected as the first camera posture;

[0007] Obtaining the relative posture between the camera and the calibration plate when the current hand frame is collected as the second camera posture;

[0008] Based on the first camera pose, the second camera pose and the gesture key point annotation information of the initial hand frame, the key point annotation of the current hand frame is performed to obtain the gesture key point annotation information of the current hand frame.

[0009] In a second aspect, the present application provides an image key point labeling device, the image key point labeling device comprising:

[0010] A first acquisition unit is used to acquire gesture key point annotation information of an initial hand frame, wherein the initial hand frame is an image including a calibration plate, and a plurality of reference points are set on the calibration plate;

[0011] A second acquisition unit, used to acquire the relative posture between the camera and the calibration plate when the initial hand frame is collected, as a first camera posture;

[0012] The second acquisition unit is further used to acquire the relative posture between the camera and the calibration plate when the current hand frame is collected, as the second camera posture;

[0013] A labeling unit is used to label the key points of the current hand frame based on the first camera posture, the second camera posture and the gesture key point labeling information of the initial hand frame to obtain the gesture key point labeling information of the current hand frame.

[0014] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it executes any one of the image key point labeling methods provided in the present application.

[0015] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program is loaded by a processor to execute the image key point labeling method.

[0016] The present application uses the relative posture between the camera and the calibration plate when the initial hand frame is captured as the first camera posture; obtains the relative posture between the camera and the calibration plate when the current hand frame is captured as the second camera posture; uses the gesture key point annotation information of the initial hand frame, the first camera posture and the second posture to annotate the key points of the current hand frame to obtain the gesture key point annotation information of the current hand frame. Since the first camera posture and the second posture can reflect the posture change of the current hand frame relative to the initial hand frame, the first camera posture and the second posture can capture the change of each gesture key point in the current hand frame relative to the initial hand frame, thereby accurately capturing the change of each gesture key point in the current hand frame. The position of each gesture key point in the previous hand frame; thus, on the first hand, since the position of each gesture key point in the initial hand frame can be used to capture the position of each gesture key point in the current hand frame, there is no need to manually annotate the gesture key points of the current hand frame after annotating the initial hand frame, thereby improving the image annotation efficiency; on the second hand, since the gesture key point annotation information of the initial hand frame, the first camera posture and the second posture can be used to accurately capture the position of each gesture key point in the current hand frame, the image annotation accuracy can be improved; on the third hand, since the current hand frame can be an actual captured image, the annotated image obtained by the embodiment of the present application has a higher degree of fit with the real image. Therefore, the embodiment of the present application can improve the image annotation efficiency and image annotation accuracy while ensuring the fit between the annotated image and the real image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a schematic block diagram of the structure of an electronic device provided in an embodiment of the present application;

[0019] Figure 2 This is a flow chart of a method for marking key points of an image provided by an embodiment of the present application;

[0020] Figure 3 is a schematic diagram illustrating the relationship between various coordinate systems in the embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a scene of a hand image in an embodiment of the present application;

[0022] Figure 5 This is a flowchart of an embodiment of step 203 in the embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the structure of an embodiment of an image key point labeling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0026] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0027] In order to enable any technician in the field to implement and use the application, the following description is given. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the embodiment of the application obscure. Therefore, the application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the application.

[0028] The embodiment of the present application provides a method, device, electronic device and computer-readable storage medium for labeling key points of an image. The device for labeling key points of an image can be integrated in an electronic device. The first acquisition can be a computer device, a server and the like.

[0029] The executor of the image key point labeling method of the embodiment of the present application may be the image key point labeling device provided in the embodiment of the present application, or an electronic device integrating the image key point labeling device, wherein the image key point labeling device may be implemented in hardware or software.

[0030] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] Figure 1 It is a schematic block diagram of the structure of an electronic device provided in an embodiment of the present application.

[0032] like Figure 1 As shown, the electronic device 100 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected via a bus 103, such as an I2C (Inter-integrated Circuit) bus.

[0033] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire electronic device 100. The processor 101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0034] Specifically, the memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0035] Those skilled in the art will understand that Figure 1 The structure shown in the figure is merely a block diagram of a partial structure related to the embodiment scheme of the present application, and does not constitute a limitation on the electronic device to which the embodiment scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0036] The processor 101 is used to run the computer program stored in the memory 102, and implement any one of the image key point labeling methods provided in the embodiments of the present application when executing the computer program. For example, the processor 101 is used to run the computer program stored in the memory 102, and can implement the following steps when executing the computer program:

[0037] Obtain gesture key point annotation information of an initial hand frame, wherein the initial hand frame is an image including a calibration plate having a plurality of reference points arranged thereon; obtain the relative pose between the camera and the calibration plate when the initial hand frame is acquired as a first camera pose; obtain the relative pose between the camera and the calibration plate when the current hand frame is acquired as a second camera pose; and annotate the current hand frame with key points based on the first camera pose, the second camera pose and the gesture key point annotation information of the initial hand frame to obtain gesture key point annotation information of the current hand frame.

[0038] It should be noted that technicians in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the following image key point labeling method embodiment, and will not be repeated here.

[0039] In the following, Figure 1 The electronic device shown in is used as an example to execute the method for labeling key points of an image, and the method for labeling key points of an image provided by the embodiment of the present application is described in detail. For the sake of simplicity and convenience of description, the execution subject will be omitted in the subsequent method embodiments. Figure 1 The scenario is only used to explain the image key point labeling method provided in the embodiment of the present application, but does not constitute a limitation on the application scenario of the image key point labeling method provided in the embodiment of the present application.

[0040] See also Figure 2 , Figure 2 201 to 203, wherein:

[0041] 201. Obtain gesture key point annotation information of an initial hand frame, wherein the initial hand frame is an image including a calibration plate, and a plurality of reference points are set on the calibration plate.

[0042] Among them, the hand frame is a hand image containing a calibration plate and gestures.

[0043] The gesture key point annotation information of the initial hand frame includes the initial frame annotation coordinates of each preset gesture key point. In some embodiments, the initial frame annotation coordinates of each preset gesture key point include the initial frame image coordinates of the preset gesture key point. In some embodiments, the initial frame annotation coordinates of each preset gesture key point include the initial frame camera coordinates of the preset gesture key point.

[0044] The initial frame image coordinates of the preset gesture key points refer to the image coordinates of the preset gesture key points in the image coordinate system of the initial hand frame.

[0045] The initial frame camera coordinates of the preset gesture key points refer to the camera coordinates of the preset gesture key points in the camera coordinate system of the initial hand frame.

[0046] In order to better understand the embodiments of the present application, some names involved in the embodiments are introduced below:

[0047] 1. Spatial three-dimensional coordinate system: A coordinate system used to describe the position of an object (such as a camera) in three-dimensional space, for example Figure 3 As shown, the three-dimensional coordinate system of space can be O w X w Y w Z w .

[0048] The spatial coordinates refer to the coordinates in the three-dimensional spatial coordinate system.

[0049] 2. Camera coordinate system: A three-dimensional coordinate system, originally defined as the light point of the camera, for example Figure 3 As shown, the camera coordinate system can be O c X c Y c Z c .

[0050] The camera coordinates refer to the coordinates in the camera coordinate system.

[0051] 3. Image coordinate system: It is a two-dimensional coordinate system. The origin is the corner point between the camera optical axis and the imaging plane. The image coordinate system and the camera coordinate system are in a perspective projection relationship. The image coordinate system is a two-dimensional coordinate system obtained by projecting the three-dimensional camera coordinate system. For example, Figure 3 As shown, the image coordinate system can be oxy, Figure 3 The dashed parallelogram box in the figure represents the image frame.

[0052] The image coordinates refer to the coordinates in the image coordinate system.

[0053] 4. Pixel coordinate system: It is a two-dimensional coordinate system. Both the pixel coordinate system and the image coordinate system are on the imaging plane. The difference is that the pixel coordinate system uses row and column values ​​to represent the coordinate values ​​of the pixel points. For example, the pixel coordinate system can be uv uv.

[0054] The pixel coordinates refer to the coordinates in the pixel coordinate system.

[0055] 5. First camera pose: The camera pose when taking the initial hand frame (i.e., the camera extrinsics of the initial gesture frame, denoted as T0).

[0056] 6. Second camera pose: The camera pose when shooting the current hand frame (i.e. the camera external parameter of the current gesture frame, denoted as T j ).

[0057] Exemplarily, when shooting, the calibration plate is used as the background, the position of the hand remains unchanged, and each frame of the image shot includes the calibration plate and the hand.

[0058] There are multiple reference points on the calibration board, for example Figure 4 As shown, taking the calibration board as the AprilTag calibration board as an example, the reference point may be the AprilTag code corner point of the AprilTag calibration board.

[0059] There are many ways to obtain the gesture key point annotation information of the initial hand frame in step 201, exemplarily including:

[0060] (1) The gesture key point annotation information includes the 2D position of each gesture key point (i.e., the image coordinates of the gesture key point); for example, the image annotation task is the 2D position of the gesture key point (i.e., the image coordinates of the gesture key point). At this time, in step 201, 2D position annotation can be performed for each gesture key point in the initial hand frame. The 2D position of the gesture key point can be determined by fine manual annotation. By receiving the manually annotated 2D position of the gesture key point, the image coordinates of each gesture key point in the initial hand frame are obtained. When the image coordinates of each gesture key point in the initial hand frame have been annotated, the gesture key point annotation information of the initial hand frame is obtained.

[0061] (2) The gesture key point annotation information includes the 3D position of each gesture key point (i.e., the camera coordinates of the gesture key point); for example, the image annotation task is the 3D position of the gesture key point (i.e., the camera coordinates of the gesture key point). In this case, in step 201, the 3D position of each gesture key point in the initial hand frame can be obtained by using tools such as a depth model or motion capture, thereby obtaining the camera coordinates of each gesture key point in the initial hand frame. When the camera coordinates of each gesture key point in the initial hand frame have been annotated, the gesture key point annotation information of the initial hand frame is obtained.

[0062] (3) The gesture key point annotation information includes the 2D position of each gesture key point (i.e., the image coordinates of the gesture key point) and the 3D position of each gesture key point (i.e., the camera coordinates of the gesture key point); for example, the image annotation task includes the 2D position of the gesture key point and the 3D position of the gesture key point. At this time, in step 201, on the one hand, 2D position annotation can be performed for each gesture key point in the initial hand frame. The 2D position of the gesture key point can be determined by fine manual annotation. By receiving the manually annotated 2D position of the gesture key point, the image coordinates of each gesture key point in the initial hand frame are obtained; on the other hand, the 3D position of each gesture key point in the initial hand frame can be obtained by using tools such as a depth model or motion capture, so as to obtain the camera coordinates of each gesture key point in the initial hand frame. When the image coordinates and camera coordinates of each gesture key point in the initial hand frame have been annotated, the gesture key point annotation information of the initial hand frame is obtained.

[0063] 202. Obtain a relative posture between the camera and the calibration plate when the initial hand frame is collected as a first camera posture.

[0064] Among them, the first camera pose refers to the camera pose when shooting the initial hand frame (that is, the camera extrinsic parameters of the initial gesture frame, that is, the first camera pose is the relative pose between the camera and the calibration plate when the initial hand frame is captured). In this embodiment, the relative pose between the camera and the calibration plate when the initial hand frame is captured is obtained as the first camera pose.

[0065] There are many ways to obtain the first camera pose in step 202, illustratively including:

[0066] (1) The first target reference point in the initial hand frame may be identified by using computer vision technology, and then the first camera pose may be solved by combining the spatial coordinates and pixel coordinates of the identified first target reference point. In this case, step 202 may specifically include the following steps 2021A to 2024A:

[0067] 2021A. Perform reference point identification on the initial hand frame to obtain the initial frame pixel coordinates of the first target reference point in the initial hand frame.

[0068] The first target reference point refers to a reference point that appears in the initial hand frame.

[0069] The initial frame pixel coordinates of the first target reference point refer to the pixel coordinates of the first target reference point in the initial hand frame.

[0070] In some embodiments, a target detection algorithm can be used to identify reference points of the initial hand frame to obtain the position of the first target reference point in the initial hand frame; the pixel coordinates of the first target reference point in the initial hand frame are used as the initial frame pixel coordinates of the first target reference point in the initial hand frame.

[0071] 2022A. Obtain the initial frame space coordinates of the first target reference point.

[0072] The initial frame space coordinates of the first target reference point refer to the space coordinates of the first target reference point when the initial hand frame is captured.

[0073] Since the position of the calibration plate remains unchanged when capturing images in this embodiment, the spatial coordinates of the same reference point remain unchanged, that is, the initial frame spatial coordinates of the same reference point are consistent with the current frame spatial coordinates. Exemplarily, before formally capturing images, the calibration plate can be placed and the position of the calibration plate relative to the three-dimensional spatial coordinate system can be set to obtain the spatial coordinates of each reference point in the calibration plate and store them in a preset database. In step 2022A, the spatial coordinates of the first target reference point are directly queried from the spatial coordinates of each reference point stored in the preset database as the initial frame spatial coordinates of the first target reference point.

[0074] 2023A. Obtain the camera intrinsic parameters of the initial hand frame.

[0075] For example, the intrinsic distortion of the camera used to capture the image can be calibrated in advance to serve as the camera intrinsic parameter of the initial hand frame. In this embodiment, the same camera is used to capture images, so the camera intrinsic parameters of each captured image frame (such as the initial hand frame, the initial hand frame) remain consistent, which can reduce the camera calibration process and parameter conversion process, and improve the image annotation efficiency to a certain extent.

[0076] 2024A. Based on the camera intrinsic parameters of the initial hand frame, the initial frame space coordinates of the first target reference point, and the initial frame pixel coordinates of the first target reference point, determine the relative pose between the camera and the calibration plate when the initial hand frame is captured as the first camera pose.

[0077] The initial frame camera coordinates of the first target reference point refer to the camera coordinates of the first target reference point in the initial hand frame.

[0078] The implementation of step 2024A is similar to that of step 2034A. For details, please refer to the relevant instructions below and will not be repeated here.

[0079] (2) In some embodiments, the calibration plate is a calibration plate provided with a positioning QR code (such as an AprilTag code). In this case, the solvepnp algorithm provided in the OpenCV library can be used to solve the initial gesture frame to obtain the relative posture between the camera and the calibration plate when the initial hand frame is captured as the first camera posture.

[0080] 203. Obtain the relative posture between the camera and the calibration plate when the current hand frame is collected, as the second camera posture.

[0081] Among them, the second camera posture refers to the camera posture when shooting the current hand frame (that is, the camera external parameters of the current gesture frame). In this embodiment, the relative posture between the camera and the calibration plate when the current hand frame is captured is obtained as the second camera posture.

[0082] Similar to the implementation of step 202, there are multiple ways to obtain the second camera pose in step 203, illustratively including:

[0083] (1) Computer vision technology can be used to first identify the second target reference point in the current hand frame, and then the second camera pose can be solved by combining the spatial coordinates and pixel coordinates of the identified second target reference point. Figure 5 As shown, step 203 may specifically include the following steps 2031A to 2034A:

[0084] 2031A. Perform reference point recognition on the current hand frame to obtain current frame pixel coordinates of a second target reference point in the current hand frame.

[0085] The second target reference point refers to a reference point that appears in the current hand frame.

[0086] The current frame pixel coordinates of the second target reference point refer to the pixel coordinates of the second target reference point in the current hand frame.

[0087] In some embodiments, a target detection algorithm can be used to identify reference points in the current hand frame to obtain the position of the second target reference point in the current hand frame; the pixel coordinates of the second target reference point in the current hand frame are used as the current frame pixel coordinates of the second target reference point in the current hand frame.

[0088] 2032A. Obtain the current frame space coordinates of the second target reference point.

[0089] The initial frame space coordinates of the second target reference point refer to the space coordinates of the second target reference point when the initial hand frame is captured.

[0090] The current frame space coordinates of the second target reference point refer to the space coordinates of the second target reference point when the current hand frame is captured.

[0091] Since the position of the calibration plate remains unchanged when capturing images in this embodiment, the spatial coordinates of the same reference point remain unchanged, that is, the initial frame spatial coordinates of the same reference point are consistent with the current frame spatial coordinates. Exemplarily, before formally capturing images, the calibration plate can be placed and the position of the calibration plate relative to the three-dimensional spatial coordinate system can be set to obtain the spatial coordinates of each reference point in the calibration plate and store them in a preset database. In step 2032A, the spatial coordinates of the second target reference point are directly queried from the spatial coordinates of each reference point stored in the preset database as the current frame spatial coordinates of the second target reference point.

[0092] 2033A. Obtain the camera intrinsic parameters of the current hand frame.

[0093] For example, the intrinsic distortion of the camera used to capture the image can be pre-calibrated to serve as the camera intrinsic parameter of the current hand frame. In this embodiment, the same camera is used to capture images, so the camera intrinsic parameters of each captured image frame (such as the initial hand frame and the current hand frame) remain consistent, which can reduce the camera calibration process and parameter conversion process, and improve the image annotation efficiency to a certain extent.

[0094] 2034A. Based on the camera intrinsic parameters of the current hand frame, the current frame space coordinates of the second target reference point, and the current frame pixel coordinates of the second target reference point, determine the relative posture between the camera and the calibration plate when the current hand frame is captured as the second camera posture.

[0095] The current frame camera coordinates of the second target reference point refer to the camera coordinates of the second target reference point in the current hand frame.

[0096] Exemplarily, the direct linear transformation method, the PnP (Perspective-n-Point) algorithm principle (such as P3P) can be used to solve the current frame pixel coordinates, the current frame space coordinates, and the camera intrinsic parameters of the current hand frame of the n (n≥3) second target reference points identified in step 2031A to obtain the relative pose between the camera and the calibration plate when the current hand frame is captured as the second camera pose.

[0097] For example, taking the P3P algorithm as an example, in step 2034, the spatial coordinates P1, P2, P3 of the second target reference points 1, 2, 3 in the three-dimensional spatial coordinate system are known, and the pixel coordinates m1, m2, m3 of the second target reference points 1, 2, 3 in the current gesture frame are known (that is, the current frame pixel coordinates of the second target reference points 1, 2, 3 are m1, m2, m3 respectively). First, according to the camera intrinsic parameters of the current hand frame, the perspective projection relationship between the camera coordinate system and the pixel coordinate system, and the current frame of the second target reference points 1, 2, 3 The pixel coordinates m1, m2, and m3 are calculated to obtain the current frame camera coordinates of the second target reference point; for example, the perspective projection relationship between the camera coordinate system and the pixel coordinate system is shown in Formula 1; the current frame pixel coordinates m1, m2, and m3 of the second target reference points 1, 2, and 3, as well as the camera intrinsic parameters of the current hand frame, are substituted into the perspective projection relationship shown in Formula 1 to obtain the current frame camera coordinates of the second target reference points 1, 2, and 3, respectively (X1, Y1, Z1), (X2, Y2, Z2), and (X3, Y3, Z3).

[0098]

[0099] In formula 1, u and v represent the pixel coordinate system O uv The coordinate value under uv, K represents the camera internal parameter, X c , Y c , Z c Represented in the camera coordinate system O c X c Y c Z c The coordinate value below.

[0100] Then, we can use the two reference points Pi and Pj in the three-dimensional coordinate system to the perspective center O c The distances xi and xj have the constraint relationship of the following formula 2. According to the spatial coordinates P1, P2, P3 of the second target reference points 1, 2, 3 in the three-dimensional coordinate system, the current frame camera coordinates (X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3) of the second target reference points 1, 2, 3 are used to construct equations, and the transformation relationship [R, t] between the camera coordinate system and the three-dimensional coordinate system is obtained by solving the equations, which is used as the relative pose between the camera and the calibration plate when the current hand frame is collected, thereby obtaining the second camera pose. Among them, [R, t] is the rotation and translation matrix.

[0101]

[0102] In formula 2, xi and xj represent the distance from two reference points Pi and Pj in the three-dimensional coordinate system to the perspective center O. c The distance, d ijRepresents the distance between two reference points Pi and Pj in the three-dimensional coordinate system of space, θ ij Represents the angle between xi and xj.

[0103] (2) In some embodiments, the calibration plate is a calibration plate provided with a positioning QR code (such as an AprilTag code). In this case, the solvepnp algorithm provided in the OpenCV library can be used to solve the current gesture frame to obtain the relative posture between the camera and the calibration plate when the current hand frame is captured, as the second camera posture.

[0104] 204. Based on the first camera posture, the second camera posture and the gesture key point annotation information of the initial hand frame, perform key point annotation on the current hand frame to obtain the gesture key point annotation information of the current hand frame.

[0105] Exemplarily, step 204 can determine the initial frame annotation coordinates of each preset gesture key point based on the gesture key point annotation information of the initial hand frame; determine the posture change of the current hand frame relative to the initial hand frame based on the first camera posture and the second camera posture; determine the current frame annotation coordinates of each preset gesture key point based on the posture change and the initial frame annotation coordinates of each preset gesture key point; and perform key point annotation on the current hand frame based on the current frame annotation coordinates of each preset gesture key point to obtain the gesture key point annotation information of the current hand key frame.

[0106] There are many ways to obtain the gesture key point annotation information of the current hand frame in step 204, exemplarily including:

[0107] (1) The gesture key point annotation information includes the 2D position of each gesture key point (i.e., the image coordinates of the gesture key point); for example, the image annotation task is the 2D position of the gesture key point (i.e., the image coordinates of the gesture key point). At this time, in step 204, the 2D position annotation can be performed for each gesture key point in the current hand frame. For example, the 2D position annotation method for each gesture key point in the current hand frame can be as shown in the following steps 2041A to 2044A:

[0108] 2041A. Determine the initial frame image coordinates of each preset gesture key point based on the gesture key point annotation information of the initial hand frame.

[0109] 2042A. Based on the first camera posture and the second camera posture, determine a posture change of the current hand frame relative to the initial hand frame.

[0110] For example, suppose the first camera position is T0 and the second camera position is T j, then the position change of the current hand frame relative to the initial hand frame is T0 j As shown in the following formula 3:

[0111] T0 j =T j *T0 -1 Formula 3

[0112] 2043A. Determine the current frame image coordinates of each preset gesture key point based on the posture change amount and the initial frame image coordinates of each preset gesture key point.

[0113] For example, suppose the change in the current hand frame’s posture relative to the initial hand frame is T0 j , the 2D position of each preset gesture key point As shown in Formula 4 below:

[0114]

[0115] In formula 4, 0≤i≤n, n is the total number of preset gesture key points, T0 j The change in the current hand frame’s pose relative to the initial hand frame, is the 2D position of the preset gesture key point i in the initial hand frame (i.e., the image coordinates of the preset gesture key point i in the initial hand frame), is the 2D position of the preset gesture key point i in the current hand frame j (ie, the image coordinates of the preset gesture key point i in the current hand frame j).

[0116] 2044A. Based on the current frame image coordinates of the preset gesture key points, key point annotation is performed on the current hand frame to obtain gesture key point annotation information of the current hand key frame.

[0117] By analogy, when the image coordinates of each preset gesture key point in the current hand frame have been labeled, the gesture key point labeling information of the current hand frame is obtained.

[0118] (2) The gesture key point annotation information includes the 3D position of each gesture key point (i.e., the camera coordinates of the gesture key point); for example, the image annotation task is the 3D position of the gesture key point (i.e., the camera coordinates of the gesture key point). At this time, in step 204, the 3D position annotation can be performed for each gesture key point in the current hand frame. For example, the 3D position annotation method for each gesture key point in the current hand frame can be as shown in steps 2041B to 2044B:

[0119] 2041B. Determine the initial frame camera coordinates of each preset gesture key point based on the gesture key point annotation information of the initial hand frame.

[0120] 2042B. Based on the first camera posture and the second camera posture, determine a posture change of the current hand frame relative to the initial hand frame.

[0121] 2043B. Determine the current frame camera coordinates of each preset gesture key point based on the posture change amount and the initial frame camera coordinates of each preset gesture key point.

[0122] For example, suppose the change in the current hand frame’s posture relative to the initial hand frame is T0 j , the 3D position of each preset gesture key point As shown in the following formula 5:

[0123]

[0124] In formula 5, 0≤i≤n, n is the total number of preset gesture key points, T0 j The change in the current hand frame’s pose relative to the initial hand frame, is the 3D position of the preset gesture key point i in the initial hand frame (i.e., the camera coordinates of the preset gesture key point i in the initial hand frame), is the 3D position of the preset gesture key point i in the current hand frame j (ie, the camera coordinates of the preset gesture key point i in the current hand frame j).

[0125] 2044B. Based on the current frame camera coordinates of the preset gesture key points, key point annotation is performed on the current hand frame to obtain gesture key point annotation information of the current hand key frame.

[0126] By analogy, when the camera coordinates of each preset gesture key point in the current hand frame have been labeled, the gesture key point labeling information of the current hand frame is obtained.

[0127] (3) The gesture key point annotation information includes the 2D position of each gesture key point (i.e., the image coordinates of the gesture key point) and the 3D position of each gesture key point (i.e., the camera coordinates of the gesture key point); for example, the image annotation task includes the 2D position of the gesture key point and the 3D position of the gesture key point. At this time, step 204 can perform 2D position and 3D position annotation for each gesture key point in the current hand frame, and step 204 can specifically include the following steps 2041C to 2045C:

[0128] 2041C. Based on the gesture key point annotation information of the initial hand frame, determine the initial frame image coordinates of each preset gesture key point and the initial frame camera coordinates of each preset gesture key point.

[0129] 2042C. Determine a change in the posture of the current hand frame relative to the initial hand frame based on the first camera posture and the second camera posture;

[0130] 2043C. Determine the current frame image coordinates of each preset gesture key point based on the posture change amount and the initial frame image coordinates of each preset gesture key point.

[0131] 2044C. Determine the current frame camera coordinates of each preset gesture key point based on the posture change amount and the initial frame camera coordinates of each preset gesture key point;

[0132] 2045C. Based on the current frame camera coordinates of the preset gesture key points and the current frame camera coordinates of the preset gesture key points, key point annotation is performed on the current hand frame to obtain gesture key point annotation information of the current hand key frame.

[0133] Therefore, on the first aspect, by determining the posture change of the current hand frame relative to the initial hand frame based on the first camera posture and the second camera posture, and using the posture change of the current hand frame relative to the initial hand frame to determine the current frame image coordinates of each preset gesture key point, the posture change between the current hand frame and the initial hand frame can be used to automatically obtain the current frame image coordinates of the preset gesture key points. Therefore, it is only necessary to annotate the key point image coordinates of the initial hand frame, and there is no need to annotate the key point image coordinates of the current hand frame again, thereby reducing the manual repeated annotation process of 2D positions and improving image annotation efficiency.

[0134] Secondly, by determining the change in posture of the current hand frame relative to the initial hand frame based on the first camera posture and the second camera posture, and using the change in posture of the current hand frame relative to the initial hand frame to determine the current frame camera coordinates of each preset gesture key point, the posture change between the current hand frame and the initial hand frame can be used to automatically obtain the current frame camera coordinates of the preset gesture key points. Therefore, it is only necessary to annotate the key point camera coordinates of the initial hand frame, and there is no need to annotate the key point camera coordinates of the current hand frame again, thereby reducing the manual repeated annotation process of 3D positions and improving image annotation efficiency.

[0135] Thirdly, by utilizing the posture changes between the current hand frame and the initial hand frame, the current frame camera coordinates of the preset gesture key points can be automatically obtained, so that the 3D positions of the image key points can be marked, avoiding the problem of being unable to mark the 3D positions of the key points when using algorithms such as optical flow.

[0136] Furthermore, in order to improve the labeling efficiency, the initial hand frame will capture as many complete gesture key points as possible, so that the initial hand frame contains as many preset gesture key points as possible, and the gesture key point labeling information of the initial hand frame contains the initial frame labeling coordinates of multiple preset gesture key points, so that when the key points in the current hand frame are subsequently labeled, each gesture key point contained in the current hand frame has the initial frame labeling coordinates, thereby improving the image labeling efficiency. However, since the camera pose of the initial hand frame is different from the camera pose of the current hand frame, in some cases there may be a situation where one or more preset gesture key points do not exist in the current hand frame. If the preset gesture key points that are not captured in the current hand frame are labeled, the labeling information will be inaccurate, which will lead to a decrease in model accuracy when the gesture key point labeling information of the current hand frame is used for model training. In order to avoid the problem of inaccurate annotation information caused by annotating preset gesture key points that are not captured in the current hand frame, exemplarily, "annotating the key points of the current hand frame based on the current frame annotation coordinates of the preset gesture key points to obtain the gesture key point annotation information of the current hand key frame" may specifically include the following steps A1 to A4:

[0137] A1. Traverse the preset gesture key points, perform detection based on the current frame annotation coordinates of the current traversal key point and the current gesture frame, and obtain the image detection result of the current traversal key point.

[0138] For example, a pre-trained key point recognition model can be used to detect the current frame annotated coordinates of the current traversal key point in the current gesture frame, and determine whether there is a gesture key point at the current frame annotated coordinates of the current traversal key point in the current gesture frame. If there is a gesture key point at the current frame annotated coordinates of the current traversal key point in the current gesture frame, the existence of the current traversal key point in the current gesture frame is used as the image detection result of the current traversal key point; otherwise, if there is no gesture key point at the current frame annotated coordinates of the current traversal key point in the current gesture frame, the absence of the current traversal key point in the current gesture frame is used as the image detection result of the current traversal key point. On the one hand, since the current frame annotated coordinates of the current traversal key point can be directly detected, there is no need to detect the entire image and avoid a large number of calculation processes; on the other hand, since it is only necessary to simply determine whether there is a gesture key point at the current frame annotated coordinates, the model accuracy requirement is relatively low, so there is no need to spend a lot of time training the key point recognition model; thereby, the annotation accuracy of the image gesture key points can be improved on the basis of reducing the amount of calculation.

[0139] A2. If the image detection result shows that the current traversal key point exists in the current gesture frame, key point marking is performed based on the current frame marking coordinates of the current traversal key point existing in the current gesture frame.

[0140] A3. If the image detection result is that the current gesture frame does not contain the current traversal key point, the current frame annotated coordinates of the current traversal key point that does not exist in the current gesture frame are filtered out.

[0141] A4. When the traversal of the preset gesture key points is completed, the gesture key point annotation information of the current hand key frame is obtained.

[0142] Therefore, the current frame annotation coordinates of each preset gesture key point are obtained by calculation, and the current frame annotation coordinates of each preset gesture key point are used to obtain the image detection result of the current traversal key point. The current traversal key point is annotated only when the image detection result shows that the current gesture frame has the current traversal key point. When the image detection result shows that the current gesture frame does not have the current traversal key point, the current frame annotation coordinates of the current traversal key point are filtered out and the current traversal key point is not annotated. This can avoid the problem of inaccurate annotation information caused by annotating the preset gesture key points that are not captured in the current hand frame, thereby improving the accuracy of the gesture key point annotation information of the current hand key frame to a certain extent.

[0143] From the above content, it can be seen that the relative posture between the camera and the calibration plate when the initial hand frame is collected is used as the first camera posture; the relative posture between the camera and the calibration plate when the current hand frame is collected is obtained as the second camera posture; the key point annotation information of the gesture key point of the initial hand frame, the first camera posture and the second posture are used to annotate the key points of the current hand frame to obtain the gesture key point annotation information of the current hand frame. Since the first camera posture and the second posture can reflect the posture change of the current hand frame relative to the initial hand frame, the first camera posture and the second posture can capture the change of each gesture key point in the current hand frame relative to the initial hand frame, so as to accurately capture the key points of each gesture. The position of the key point in the current hand frame; therefore, on the first hand, since the position of each gesture key point in the initial hand frame can be used to capture the position of each gesture key point in the current hand frame, there is no need to manually annotate the gesture key points of the current hand frame after annotating the initial hand frame, thereby improving the efficiency of image annotation; on the second hand, since the gesture key point annotation information of the initial hand frame, the first camera posture and the second posture can be used to accurately capture the position of each gesture key point in the current hand frame, the image annotation accuracy can be improved; on the third hand, since the current hand frame can be an actual captured image, the annotated image obtained by the embodiment of the present application has a higher degree of fit with the real image. Therefore, the embodiment of the present application can improve the image annotation efficiency and image annotation accuracy while ensuring the fit between the annotated image and the real image.

[0144] In addition, in order to better implement the image key point labeling method in the embodiment of the present application, based on the image key point labeling method, the embodiment of the present application also provides an image key point labeling device, such as Figure 6 FIG. 1 is a schematic diagram of a structure of an embodiment of an image key point labeling device provided in an embodiment of the present application. The image key point labeling device 600 includes:

[0145] A first acquisition unit 601 is used to acquire gesture key point annotation information of an initial hand frame, wherein the initial hand frame is an image including a calibration plate, and a plurality of reference points are set on the calibration plate;

[0146] A second acquisition unit 602 is used to acquire the relative posture between the camera and the calibration plate when the initial hand frame is collected, as a first camera posture;

[0147] The second acquisition unit 602 is further used to acquire the relative posture between the camera and the calibration plate when the current hand frame is collected, as the second camera posture;

[0148] The labeling unit 603 is used to label the key points of the current hand frame based on the first camera posture, the second camera posture and the gesture key point labeling information of the initial hand frame to obtain the gesture key point labeling information of the current hand frame.

[0149] In some embodiments, the second acquiring unit 602 is specifically configured to:

[0150] Performing reference point recognition on the current hand frame to obtain current frame pixel coordinates of a second target reference point in the current hand frame;

[0151] Obtaining the current frame space coordinates of the second target reference point;

[0152] Obtaining the camera intrinsic parameters of the current hand frame;

[0153] Based on the camera intrinsic parameters of the current hand frame, the current frame space coordinates of the second target reference point, and the current frame pixel coordinates of the second target reference point, the relative pose between the camera and the calibration plate when the current hand frame is captured is determined as the second camera pose.

[0154] In some embodiments, the second acquiring unit 602 is specifically configured to:

[0155] Performing reference point recognition on the initial hand frame to obtain initial frame pixel coordinates of a first target reference point in the initial hand frame;

[0156] Acquire the initial frame space coordinates of the first target reference point;

[0157] Obtaining camera intrinsic parameters of the initial hand frame;

[0158] Based on the camera intrinsic parameters of the initial hand frame, the initial frame space coordinates of the first target reference point, and the initial frame pixel coordinates of the first target reference point, the relative pose between the camera and the calibration plate when the initial hand frame is captured is determined as the first camera pose.

[0159] In some embodiments, the labeling unit 603 is specifically used to:

[0160] Based on the gesture key point annotation information of the initial hand frame, determining the initial frame annotation coordinates of each preset gesture key point;

[0161] Determining a change in the posture of the current hand frame relative to an initial hand frame based on the first camera posture and the second camera posture;

[0162] Determining the current frame annotation coordinates of each preset gesture key point based on the posture change amount and the initial frame annotation coordinates of each preset gesture key point;

[0163] Based on the current frame annotation coordinates of the preset gesture key points, the key points of the current hand frame are annotated to obtain gesture key point annotation information of the current hand key frame.

[0164] In some embodiments, the annotated coordinates include camera coordinates, and the annotating unit 603 is specifically used to:

[0165] Based on the posture change amount and the initial frame camera coordinates of the preset gesture key points, the current frame camera coordinates of the preset gesture key points are determined.

[0166] In some embodiments, the annotated coordinates include image coordinates, and the annotating unit 603 is specifically used to:

[0167] Based on the posture change amount and the initial frame image coordinates of the preset gesture key points, the current frame image coordinates of the preset gesture key points are determined.

[0168] In some embodiments, the labeling unit 603 is specifically used to:

[0169] Traversing the preset gesture key points, performing detection based on the current frame annotated coordinates of the currently traversed key point and the current gesture frame, and obtaining an image detection result of the currently traversed key point;

[0170] If the image detection result shows that the current traversal key point exists in the current gesture frame, the current traversal key point is marked as a key point based on the current frame marked coordinates of the current traversal key point; or, if the image detection result shows that the current traversal key point does not exist in the current gesture frame, the current frame marked coordinates of the current traversal key point are filtered out;

[0171] When the traversal of the preset gesture key points is completed, the gesture key point annotation information of the current hand key frame is obtained.

[0172] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above units can refer to the previous image key point labeling method embodiment, which will not be repeated here.

[0173] A person of ordinary skill in the art will appreciate that all or part of the steps in the above-mentioned method for labeling key points in an image may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0174] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. The computer programs can be loaded by a processor to execute any image key point labeling method provided in the embodiment of the present application.

[0175] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0176] In the above-mentioned image key point labeling device, computer-readable storage medium, and electronic device embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described image key point labeling device, computer-readable storage medium, electronic device and its corresponding units can refer to the description of the image key point labeling method in the above embodiment, and will not be repeated here.

[0177] The above is a detailed introduction to an image key point labeling method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for labeling key points of an image, characterized in that: The method comprises: Acquire gesture key point annotation information of an initial hand frame, wherein the initial hand frame is an image including a calibration plate, and a plurality of reference points are set on the calibration plate; Obtaining the relative posture between the camera and the calibration plate when the initial hand frame is collected as the first camera posture; Obtaining the relative posture between the camera and the calibration plate when the current hand frame is collected as the second camera posture; Based on the first camera pose, the second camera pose and the gesture key point annotation information of the initial hand frame, the key point annotation of the current hand frame is performed to obtain the gesture key point annotation information of the current hand frame.

2. The image key point labeling method according to claim 1, characterized in that: The step of obtaining the relative posture between the camera and the calibration plate during the current hand frame acquisition as the second camera posture includes: Performing reference point recognition on the current hand frame to obtain current frame pixel coordinates of a second target reference point in the current hand frame; Obtaining the current frame space coordinates of the second target reference point; Obtaining the camera intrinsic parameters of the current hand frame; Based on the camera intrinsic parameters of the current hand frame, the current frame space coordinates of the second target reference point, and the current frame pixel coordinates of the second target reference point, the relative pose between the camera and the calibration plate when the current hand frame is captured is determined as the second camera pose.

3. The image key point labeling method according to claim 1, characterized in that: The step of obtaining the relative posture between the camera and the calibration plate during the initial hand frame acquisition as the first camera posture includes: Performing reference point recognition on the initial hand frame to obtain initial frame pixel coordinates of a first target reference point in the initial hand frame; Acquire the initial frame space coordinates of the first target reference point; Obtaining camera intrinsic parameters of the initial hand frame; Based on the camera intrinsic parameters of the initial hand frame, the initial frame space coordinates of the first target reference point, and the initial frame pixel coordinates of the first target reference point, the relative pose between the camera and the calibration plate when the initial hand frame is captured is determined as the first camera pose.

4. The image key point labeling method according to claim 1, characterized in that: The step of performing key point annotation on the current hand frame based on the first camera pose, the second camera pose and the gesture key point annotation information of the initial hand frame to obtain the gesture key point annotation information of the current hand frame includes: Based on the gesture key point annotation information of the initial hand frame, determining the initial frame annotation coordinates of each preset gesture key point; Determining a change in the posture of the current hand frame relative to an initial hand frame based on the first camera posture and the second camera posture; Determining the current frame annotation coordinates of each preset gesture key point based on the posture change amount and the initial frame annotation coordinates of each preset gesture key point; Based on the current frame annotation coordinates of the preset gesture key points, the key points of the current hand frame are annotated to obtain gesture key point annotation information of the current hand key frame.

5. The method for labeling key points of an image according to claim 4, characterized in that: The annotated coordinates include camera coordinates, and determining the current frame annotated coordinates of each preset gesture key point based on the posture change amount and the initial frame annotated coordinates of each preset gesture key point includes: Based on the posture change amount and the initial frame camera coordinates of the preset gesture key points, the current frame camera coordinates of the preset gesture key points are determined.

6. The method for labeling key points of an image according to claim 4, characterized in that: The annotated coordinates include image coordinates, and determining the current frame annotated coordinates of each preset gesture key point based on the posture change amount and the initial frame annotated coordinates of each preset gesture key point includes: Based on the posture change amount and the initial frame image coordinates of the preset gesture key points, the current frame image coordinates of the preset gesture key points are determined.

7. The method for labeling key points of an image according to claim 4, characterized in that: The step of labeling the key points of the current hand frame based on the current frame labeling coordinates of the preset gesture key points to obtain the gesture key point labeling information of the current hand key frame includes: Traversing the preset gesture key points, performing detection based on the current frame annotated coordinates of the currently traversed key point and the current gesture frame, and obtaining an image detection result of the currently traversed key point; If the image detection result shows that the current traversal key point exists in the current gesture frame, key point annotation is performed based on the current frame annotation coordinates of the current traversal key point that exists in the current gesture frame; or, if the image detection result shows that the current traversal key point does not exist in the current gesture frame, the current frame annotation coordinates of the current traversal key point that does not exist in the current gesture frame are filtered out; When the traversal of the preset gesture key points is completed, the gesture key point annotation information of the current hand key frame is obtained.

8. An image key point labeling device, characterized in that: The image key point marking device comprises: A first acquisition unit is used to acquire gesture key point annotation information of an initial hand frame, wherein the initial hand frame is an image including a calibration plate, and a plurality of reference points are set on the calibration plate; A second acquisition unit, used to acquire the relative posture between the camera and the calibration plate when the initial hand frame is collected, as a first camera posture; The second acquisition unit is further used to acquire the relative posture between the camera and the calibration plate when the current hand frame is collected, as the second camera posture; A labeling unit is used to label the key points of the current hand frame based on the first camera posture, the second camera posture and the gesture key point labeling information of the initial hand frame to obtain the gesture key point labeling information of the current hand frame.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method for labeling key points of an image as claimed in any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the image key point labeling method according to any one of claims 1 to 7.

Images