Image annotation method and related device and equipment

Abstract

The present application discloses a method for labeling a picture and related devices and equipment, wherein the method for labeling a picture includes: determining the number and type of points to be labeled based on the picture to be labeled; labeling the picture to be labeled accordingly based on the number and type of points to be labeled to obtain multiple labeling points; selecting a target labeling point from the multiple labeling points, and connecting the target labeling point with at least one other labeled labeling point to obtain the labeling information of the picture based on the labeling points and the connecting lines. The above scheme can improve the labeling effect on the labeled picture by labeling the picture and connecting the various labeling points, thereby improving the practicality of the labeling information.

Description

Technical Field

[0001] The present application relates to the technical field of image annotation, and in particular to an image annotation method and related devices and equipment. Background Art

[0002] With the rapid development of Internet technology, computer vision has achieved remarkable achievements in the field of artificial intelligence deep learning. In addition to facial key point recognition, the application scenarios and related technologies of body recognition based on human shape and posture are also making continuous breakthroughs.

[0003] Among them, labeling the characteristic points or joint points of the human body or human skeleton is crucial for the relevant training of virtual human simulation. At present, the labeling tools in the industry generally only label the points on the picture. This method has poor labeling effect, low practicality and low labeling efficiency. Summary of the invention

[0004] The present application provides a method for labeling an image and related devices and equipment to solve the problem of poor image labeling effect in the prior art.

[0005] The present application provides a method for annotating an image, comprising: determining the number and type of points to be annotated based on the image to be annotated; annotating the image to be annotated accordingly based on the number and type of points to be annotated to obtain a plurality of annotation points; selecting a target annotation point from the plurality of annotation points, and connecting the target annotation point with at least one other annotated annotation point to obtain annotation information of the image based on the annotation points and the connecting lines.

[0006] Among them, the step of selecting a target marking point from multiple marking points and connecting the target marking point with at least one other marked marking point includes: connecting the target marking point with the other marking points based on a preset connection rule; establishing a corresponding relationship between the target marking point, other marking points, and the connection between the target marking point and the other marking points to obtain the target marking point, other marking points, and the combination of the target marking point and the other marking points.

[0007] Among them, after the step of selecting a target marking point from multiple marking points and connecting the target marking point with at least one other marked marking point, the following step is included: displaying the connection line between the target marking point and the other marking points based on a preset color and a preset transparency.

[0008] Among them, after establishing a corresponding relationship between the target marking point, other marking points and the connection line between the target marking point and the other marking points to obtain the target marking point, other marking points and the combination of the target marking point and the other marking points, the step includes: using the combination to delete the target marking point and the connection line corresponding to the target marking point.

[0009] The image marking method further includes: storing in real time the position information and type information of each marking point and the color and transparency of each connecting line to facilitate the cancellation of the marking.

[0010] Among them, the step of annotating the picture to be annotated accordingly based on the number and type of points to be annotated to obtain multiple annotation points includes: identifying annotation points of the picture to be annotated through the picture annotation model based on the number and type of points to be annotated, and obtaining initial annotation information of at least one pre-annotated point; marking at least one pre-annotated point on the picture to be annotated using the initial annotation information of at least one pre-annotated point; and annotating the picture to be annotated again to mark the remaining annotation points on the picture to be annotated, and obtaining multiple annotation points.

[0011] Among them, the step of identifying annotation points of the picture to be labeled through the picture annotation model based on the number and type of points to be labeled, and obtaining initial annotation information of at least one pre-annotated point includes: identifying annotation points of the picture to be labeled through the picture annotation model to obtain initial coordinate information and initial type information of each pre-annotated point; configuring the initial coordinate information and initial type information of each pre-annotated point to obtain label features of each pre-annotated point; the step of marking at least one pre-annotated point on the picture to be labeled using the initial annotation information of at least one pre-annotated point includes: marking each pre-annotated point on the picture to be labeled using the label features of each pre-annotated point through the picture annotation model.

[0012] Among them, the step of performing secondary annotation on the image to be annotated to mark the remaining annotation points on the image to be annotated to obtain multiple annotation points includes: performing secondary annotation on the image to be annotated to mark the remaining annotation points on the image to be annotated, and determining the type of each remaining annotation point; and / or moving the position of at least one pre-annotated point to obtain the annotation points.

[0013] Among them, the step of obtaining the annotation information of the image based on the annotation points and the connecting lines includes: traversing each annotation point and each connecting line on the annotated image to obtain the determined coordinate information, determined type information of each annotation point, and the trajectory information and connection information of each connecting line; using the determined coordinate information, determined type information of each annotation point, and the trajectory information and connection information of each connecting line as the annotation information of the image.

[0014] The present application also provides a device for labeling an image, including: a determination module, used to determine the number and type of points to be labeled based on the image to be labeled; a labeling module, used to label the image to be labeled accordingly based on the number and type of points to be labeled, so as to obtain multiple labeling points; a connection module, used to select a target labeling point from the multiple labeling points, and connect the target labeling point with at least one other labeled labeling point, so as to obtain the labeling information of the image based on the labeling points and the connection lines.

[0015] The present application also provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement any of the above-mentioned image labeling methods.

[0016] The present application also provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, any of the above-mentioned image labeling methods is implemented.

[0017] In the above scheme, the present application first determines the number and type of points to be annotated based on the picture to be annotated, and then annotates the picture to be annotated accordingly based on the number and type of points to be annotated to obtain multiple annotation points, selects a target annotation point from the multiple annotation points, and connects the target annotation point with at least one other annotated annotation point to obtain the annotation information of the picture based on the annotation points and the connecting lines, thereby improving the annotation effect on the annotated picture by connecting the annotation points, and enhancing the visual correspondence between the annotation points on the picture, thereby improving the practicality of the annotation information when the annotation information is applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of an embodiment of a method for labeling pictures of the present application;

[0019] Figure 2 It is a flowchart of another embodiment of the method for labeling pictures of the present application;

[0020] Figure 3 yes Figure 2 A schematic diagram of an embodiment in which points to be marked in a selected area are being marked;

[0021] Figure 4 yes Figure 2 Schematic diagram of an embodiment of Figure 1 after the marking in the embodiment is completed;

[0022] Figure 5 It is a schematic diagram of the framework of an embodiment of a device for labeling pictures of the present application;

[0023] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0024] Figure 7 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0026] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0027] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects. There can be three kinds of relationships. For example, A and / or B can be: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally means that the associated objects before and after are in an "or" relationship. In addition, "more" in this article means two or more than two.

[0028] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for labeling pictures of the present application. Specifically, the following steps may be included:

[0029] Step S11: Determine the number and type of points to be annotated based on the image to be annotated.

[0030] In a specific application scenario, the images to be annotated may be multiple images with the same characteristic objects, for example, 100 images containing human body parts or 200 images containing animal skeleton parts, etc. The specific number of images to be annotated and the characteristic objects may be determined according to actual conditions and are not limited here.

[0031] In a specific application scenario, the number and type of points to be labeled can be determined based on all features contained in the feature object of the picture to be labeled. Among them, the number of points to be labeled of the picture to be labeled is determined based on the number of all features of the feature object. In a specific application scenario, when the feature object on multiple pictures to be labeled is a human body, the number and type of points to be labeled can be determined based on all joint features of the human body, that is, the number of points to be labeled is the number of all joint features of the human body. Among them, the number of points to be labeled is not set according to the features contained in the range of the human body displayed on a single picture to be labeled. For example: when the human body part on the picture to be labeled is only the upper body, the range of all joint features of the human body is the entire human body, not just the upper body.

[0032] In a specific application scenario, when the feature object of multiple images to be annotated is a human body, the number and type of the points to be annotated can be determined based on all the joint feature points of the human body. For example, when there are 20 joint feature points of the human body, and the types are "nose", "under the neck", "right shoulder", "right elbow", "right wrist", "left shoulder", "left elbow", "left wrist", "right hip", "right knee", "right ankle", "left hip", "left knee", "left ankle", "right eye", "left eye", "right ear", "left ear", "left palm", and "right palm", the number of points to be annotated is also 20, and the types are "nose", "under the neck", "right shoulder", "right elbow", "right wrist", "left shoulder", "left elbow", "left wrist", "right hip", "right knee", "right ankle", "left hip", "left knee", "left ankle", "right eye", "left eye", "right ear", "left ear", "left palm", and "right palm". The number of features of a specific feature object can be determined manually or by a feature extraction model, which is not limited here.

[0033] In a specific application scenario, the images to be annotated may also be multiple images with the same characteristic scene, for example, 100 architectural images. The specific number of images to be annotated and the characteristic scene can be determined according to the actual situation and are not limited here. The number and type of points to be annotated can be determined based on the characteristics of the characteristic scene.

[0034] Step S12: annotate the image to be annotated based on the number and type of the points to be annotated to obtain a plurality of annotated points.

[0035] Based on the number and type of the points to be annotated obtained in the previous step, each image to be annotated is annotated accordingly to obtain multiple corresponding annotation points on each image to be annotated.

[0036] In a specific application scenario, after determining 20 points to be annotated on the entire human body based on the number of human body features, when annotating the target image to be annotated that only contains part of the human body, only the part of the human body that is intuitively displayed in the target image to be annotated is annotated, and the part of the human body that is not intuitively displayed in the target image to be annotated is not annotated, thereby obtaining multiple annotated points of the target image to be annotated. For example, when the target image to be annotated only contains the upper body, the target image to be annotated is annotated based on the number and type of points to be annotated in the upper body of the human body, thereby obtaining multiple annotated points.

[0037] Step S13: Select a target annotation point from the multiple annotation points, and connect the target annotation point with at least one other annotated annotation point to obtain annotation information of the image based on the annotation points and the connection lines.

[0038] After the target image to be annotated is annotated and multiple annotation points are obtained, a target annotation point is selected from the multiple annotation points, and a line is connected between the target annotation point and at least one other annotated annotation point to obtain annotation information of the image based on the annotation points and the lines.

[0039] In a specific application scenario, after marking all the annotation points, one of the target annotation points can be selected, and the target annotation point can be connected with other annotated annotation points until all the annotation points are connected with corresponding lines. In another specific application scenario, during the annotation process, after marking each target annotation point, the target annotation point can be connected with other annotated annotation points until all the annotation points are marked. By connecting the lines, a visual correspondence is established between the annotation points, thereby improving the annotation effect on the annotated image, and then improving the practicality of the annotation information when the annotation information is applied.

[0040] After the marking and connecting are completed, the marking information of the image and the marked image are obtained based on the marked points and connecting lines.

[0041] Through the above method, the image annotation method of this embodiment first determines the number and type of points to be annotated based on the image to be annotated, and then annotates the image to be annotated accordingly based on the number and type of points to be annotated to obtain multiple annotation points, selects a target annotation point from the multiple annotation points, and connects the target annotation point with at least one other annotated annotation point to obtain the annotation information of the image based on the annotation points and the connecting lines, thereby improving the annotation effect on the annotated image by connecting the annotation points, and enhancing the visual correspondence between the annotation points on the image, thereby improving the practicality of the annotation information when the annotation information is applied.

[0042] See also Figure 2 , Figure 2 This is a flow chart of another embodiment of the method for labeling pictures of the present application. This embodiment will be described by taking the common feature of the pictures to be labeled as the human body structure. In other embodiments, the common feature of the pictures to be labeled can be any other structure or feature, which is not limited here. Specifically, the following steps can be included:

[0043] The annotation method of this embodiment is applied to an annotation tool of an annotation platform. After acquiring the image to be annotated, the annotation platform annotates the image to be annotated through the annotation tool.

[0044] Step S21: Determine the number and type of points to be annotated based on the image to be annotated.

[0045] The number and type of points to be annotated are determined based on the characteristics of the human body structure common to the acquired multiple images to be annotated. In this embodiment, the type of the point to be annotated refers to the part type of the point to be annotated.

[0046] In a specific application scenario, the features of the human body structure determined manually are ["nose", "under the neck", "right shoulder", "right elbow", "right wrist", "left shoulder", "left elbow", "left wrist", "right hip", "right knee", "right ankle", "left hip", "left knee", "left ankle", "right eye", "left eye", "right ear", "left ear", "left palm", "right palm"], then the number of fixed points to be marked that can be determined in this step is 20, and their types are ["nose", "under the neck", "right shoulder", "right elbow", "right wrist", "left shoulder", "left elbow", "left wrist", "right hip", "right knee", "right ankle", "left hip", "left knee", "left ankle", "right eye", "left eye", "right ear", "left ear", "left palm", "right palm"].

[0047] After determining the number and type of points to be annotated, the annotation labels of multiple points to be annotated are determined based on the number and type of points to be annotated, and multiple annotation labels can be placed in the selection area of ​​the annotation tool in the order of human body structure. In a specific application scenario, a certain annotation label can be selected for annotation. After annotation, the next unlabeled annotation label is automatically selected based on the order of human body structure for easy marking. Among them, the automatic selection method can be performed by modulus operation or other methods. When the modulus operation is selected for automatic selection, the following code [remainPoints[(index+1)%remainPoints.length] can be used.

[0048] See also Figure 3 , Figure 3 yes Figure 2 A schematic diagram of an embodiment in which points to be marked in a selected area are being marked.

[0049] A plurality of annotation labels are arranged in sequence in the selection area 30, wherein the plurality of annotation labels correspond to the points to be annotated one by one. In this embodiment, the selection area 30 includes an annotated annotation label 31, an unannotated annotation label 32, and an annotated annotation label 33. After the annotated annotation label 33 is annotated, the selection area 30 automatically selects the next unannotated annotation label 32 for annotation.

[0050] In this embodiment, when annotating the image to be annotated, you can choose to annotate all the points to be annotated before the annotating is finished, or you can choose to annotate only some of the points to be annotated before the annotating is finished. The specific setting can be based on the actual situation and is not limited here.

[0051] Step S22: Based on the number and type of the points to be annotated, the image to be annotated is identified by using the image annotation model to obtain initial annotation information of at least one pre-annotated point, and at least one pre-annotated point is annotated on the image to be annotated using the initial annotation information of at least one pre-annotated point.

[0052] Since there may be many annotation points to be drawn on the image to be annotated, the annotation process in this embodiment can be performed by using Canvas to draw the image to be annotated and the annotation points on the web front end, thereby reducing the problems of front-end performance bottlenecks or page freezes. Specifically, Canvas is used to create a canvas, and then the image to be annotated is drawn on the canvas, and then Canvas is used to draw the points to be annotated on the image to be annotated on the canvas.

[0053] Before manually annotating the image to be annotated, the image to be annotated may be first identified based on the number and type of points to be annotated using an image annotation model to obtain initial annotation information of at least one pre-annotated point, and at least one pre-annotated point may be annotated on the image to be annotated using the initial annotation information of at least one pre-annotated point. The initial annotation information may include initial coordinate information and initial type information of the pre-annotated point on the image. The initial type information includes: none, occluded, and visible, which is different from the type of the annotation point.

[0054] The image annotation model is used to identify the annotation points of the image to be annotated, and the initial coordinate information and initial type information of each pre-annotated point are obtained; the initial coordinate information and initial type information of each pre-annotated point are configured to obtain the label features of each pre-annotated point; and the label features of each pre-annotated point are used to annotate each pre-annotated point on the image to be annotated. Among them, the image annotation model is an image annotation model obtained by pre-training based on image annotation, which can be a deep neural network model or other models, which are not limited here.

[0055] In this step, a JSON (JavaScript Object Notation, Java script object annotation, is a lightweight data exchange format) is defined to store the coordinate information and type information of the pre-annotated points, which provides an interface for the auxiliary annotation of the image annotation model. Specifically, the outer layer of the JSON configuration is an object, the key name of the inner layer is the name of the point, and the key value is an array of length 3. The values ​​in this array represent the horizontal coordinate relative to the left boundary of the image to be annotated, the vertical coordinate relative to the upper boundary of the image to be annotated, and the type of the pre-annotated point in order. When the initial type information of the pre-annotated point is none, the corresponding data is 0; when the initial type information of the pre-annotated point is occluded, the corresponding data is 1; when the initial type information of the pre-annotated point is visible, the corresponding data is 2. (Invisible is 0, occluded is 1, and visible is 2). Examples of configured label features are as follows:

[0056] {"right wrist": [456.12207, 915.48486, 2], "under the neck": [445.7528, 991.6262, 2], "nose": [439.641, 1065.5913, 2]}

[0057] Take "right wrist": [456.12207, 915.48486, 2] as an example for explanation. "right wrist" is the name and type of the pre-annotated point, 456.12207, 915.48486 are the coordinates of the pre-annotated point, and 2 is the initial type information of the pre-annotated point, which is visible.

[0058] The label feature of this embodiment is that the labelFeature of the labeling tool and the final labeling information share a field. Before the second labeling, the image labeling model is used for pre-labeling, and the pre-labeled label feature information is filled into the LabelFeature field. After the second labeling is modified and updated, the new labeling information will replace the old labelFeature. Or, it is divided into two fields to distinguish the pre-labeled labeling information and the final labeling information.

[0059] In a specific application scenario, after obtaining the label features of each pre-annotated point through the image annotation model, the label features can be traversed, and the pre-annotated points can be drawn onto the image to be annotated in the canvas through Canvas.

[0060] In a specific application scenario, the image annotation model uses the label features of each pre-annotated point to mark each pre-annotated point on the image to be annotated. This can be achieved through the following code:

[0061]

[0062] The labelFeature data structure is the data structure of the pre-annotated label feature information. Parse labelFeature to obtain the name, horizontal coordinate, vertical coordinate and point type of the pre-annotated point. Calibrate and draw points based on the above coordinates, and then draw the annotated points on the image to be annotated. The coordinate value is divided by ratio to calibrate the scale and position of the image on the canvas during the annotation process.

[0063] In a specific application scenario, the number of pre-annotated points annotated by the image annotation model may be less than the number of points to be annotated obtained in the above steps. Since the accuracy of the image annotation model is affected by the training data and the number of iterations, the pre-annotated points annotated by the image annotation model may be inaccurate. Therefore, in this embodiment, the image pre-annotated by the image annotation model needs to be annotated again to improve the accuracy and precision of the image annotation.

[0064] In a specific application scenario, when a pre-annotated point is marked on the image to be annotated, an instance of the name of the pre-annotated point will be created to intuitively display the specific information of the pre-annotated point. For example, when marking the left palm point, in addition to drawing a dot at the target position, a corresponding text instance will be created to display the text "left palm" to make the information of the annotation point intuitive. The subsequent secondary annotation will also create an instance of the name of the pre-annotated point for display.

[0065] Step S23: performing secondary annotation on the image to be annotated, so as to mark the remaining annotation points on the image to be annotated, and obtain a plurality of annotation points.

[0066] Due to manual annotation, the image to be annotated needs to be displayed on the screen or visible area for manual viewing and annotation. Therefore, before the image to be annotated is manually annotated for the second time, the image to be annotated is first drawn to the canvas using Canvas (a drawing tool) on the web front end. Since the display area of ​​the screen or visible area is fixed and limited, in order to enable the image to be annotated to match the screen or visible area after initialization and reduce the occurrence of images out of the frame, it is necessary to calculate the proportion of the image to be annotated. In a specific application scenario, if the aspect ratio of the image to be annotated is larger than the visible area, the image to be annotated is scaled proportionally to a width equal to the width of the visible area, otherwise the height is scaled proportionally to the height of the visible area. The scaling function in this application scenario can be implemented by the following code:

[0067]

[0068]

[0069] The image to be labeled is manually labeled for a second time to mark the remaining annotation points on the image to be labeled, thereby obtaining multiple annotation points, that is, all the annotation points that can be marked by the human body structure displayed on the image to be labeled.

[0070] In the secondary annotation process, since this step is manual annotation, the person clicks a certain position on the canvas through Canvas to directly draw the annotation point on the image to be annotated. The secondary annotation method of drawing the annotation point is directly done manually, which is different from the method of first obtaining the annotation label of the pre-annotated point in the pre-annotation method and then drawing based on the annotation label.

[0071] In a specific application scenario, since the pre-annotation points marked by the image annotation model are affected by their accuracy, the pre-annotation points marked by the image annotation model may be inaccurate and incomplete. Therefore, the image to be labeled is annotated again through manual annotation. Specifically, the image to be labeled can be annotated again to mark the remaining annotation points other than the pre-annotation points on the image to be labeled, and determine the final determined type information of each remaining annotation point; and / or, when the position of the pre-annotation point is inaccurate, the position of at least one inaccurate pre-annotation point is moved through secondary annotation to obtain multiple determined annotation points required to be marked on the image to be labeled.

[0072] In a specific application scenario, when the initial coordinate information and initial type information of the pre-annotated point are inaccurate, they are modified through secondary annotation in this step to obtain the final determined coordinate information and determined type information of the pre-annotated point.

[0073] In a specific application scenario, in step S22 and step S23, in the process of marking the pre-marked points and the marked points on the to-be-marked picture, the marked points can be connected based on the preset connection rule. The preset connection rule can be set by the arrangement order of the human body structure. In this embodiment, the connection rule between the marked points is determined based on the number and type of the to-be-marked points.

[0074] In a specific application scenario, the connection rule between the annotation points may be as follows: ["nose", "right eye"], ["nose", "left eye"], ["right eye", "right ear"], ["left eye", "left ear"], ["nose", "under the neck"], ["under the neck", "right shoulder"], ["under the neck", "left shoulder"], ["right shoulder", "right elbow"], ["right elbow", "right wrist"], ["left shoulder", "left elbow"], ["left elbow", "left wrist"], ["right wrist", "right palm"], ["left wrist", "left palm"], ["under the neck", "right hip"], ["right hip", "right knee"], ["right knee", "right ankle"], ["under the neck", "left hip"], ["left hip", "left knee"], ["left knee", "left ankle"]. In this embodiment, an annotation point may be connected to more than one other annotation point. For example, a certain annotation point may be connected to five other annotation points. In other embodiments, for example, in facial feature annotation, there may only be a situation where a marked point is connected to a single other marked point, which is specifically set according to the specific features of the feature structure and is not limited here.

[0075] In a specific application scenario, before annotating the image to be annotated, the color of the connection line can be set to further highlight the visualization of the annotation result. In a specific application scenario, the color of the connection line can be set to red by default. In another specific application scenario, the function of customizing the line color can be provided, which can be configured through the following data structure: {"Under the neck - right shoulder": "(255, 0, 85, 0.65)", "Under the neck - left shoulder": "(255, 0, 0, 0.65)", "Right shoulder - right elbow": "(255, 85, 0, 0.65)", "Right elbow - right wrist": "(255, 170, 0, 0.65)", "Left shoulder - left elbow": "(255, 255, 0, 0.65)", "Left elbow - left wrist": "(170, 255, 0, 0.65)", "Under the neck - right hip": "(85, 255, 0, 0.65)", "Right hip - right knee": "(0, 255, 0, 0.65)", "Right knee - right ankle": "(0, 255, 85, 0.65)", "Under the neck - left hip ”: "(0, 255, 170, 0.65)", "Left Hip - Left Knee": "(0, 255, 255, 0.65)", "Left Knee - Left Ankle": "(0, 170, 255, 0.65)", "Nose - Under Neck": "(0, 85, 255, 0.65)", "Nose - Right Eye": "(0, 0, 255, 0.65)", "Right Eye - Right Ear": "(255, 0, 17 , 0, 0.65)","Nose-Left Eye": "(170, 0, 255, 0.65)","Left Eye-Left Ear": "(255, 0, 255, 0.65)","Right Hip-Left Hip": "(85, 0, 255, 0.65)","Right Wrist-Right Palm": "(255, 255, 0, 0.65)","Left Wrist-Left Palm": "(0, 255, 0, 0.65)"}.

[0076] Here, ["left wrist - left palm": "(0, 255, 0, 0.65)"] is used as an example for explanation. Other data structures are similar to this example and are not described in detail. "Left wrist - left palm" means connecting the marked point of the left wrist with the marked point of the left palm. The first three digits 0, 255, 0 in "(0, 255, 0, 0.65)" represent the RGB color of the line, and the last digit 0.65 represents the transparency of the line. The RGB color and transparency of the line can be set based on the actual situation. This is only for illustration and not for limitation.

[0077] In this embodiment, the connection line can be filled with color by using the map coloring tool, and the logic of customizing the connection line color is specifically implemented by the following code.

[0078]

[0079]

[0080] When the marked points are connected based on the preset connection rule, the connection between the marked points is displayed based on the preset color and the preset transparency.

[0081] In a specific application scenario, after marking all the annotation points, one of the target annotation points can be selected, and the target annotation point can be connected with other annotated annotation points until all the annotation points are connected with corresponding lines. In another specific application scenario, during the annotation process, after marking each target annotation point, the target annotation point can be connected with other annotated annotation points until all the annotation points are marked. By connecting the lines, a visual correspondence is established between the annotation points, thereby improving the annotation effect on the annotated image, and then improving the practicality of the annotation information when the annotation information is applied.

[0082] In a specific application scenario, after the target annotation point is connected with other annotation points, a corresponding relationship is established between the target annotation point, other annotation points, and the connection between the target annotation point and other annotation points to obtain the target annotation point, other annotation points, and the combination of the target annotation point and other annotation points. Among them, when a target annotation point has a connection with multiple other target mark points, a corresponding relationship can be established between the target annotation point and all other annotation points and the connection for combination. By combining the annotation points and the connection, the subsequent adjustment steps can be performed on a whole, such as defining some attributes for the group or associating some object instances, scaling, deleting, etc., to improve the annotation efficiency.

[0083] When the target annotation point is wrong and needs to be deleted, you can directly use the above combination to delete the target annotation point, the instance of the target annotation point, and the line corresponding to the target annotation point at one time. Otherwise, there will be errors in the subsequent interactions such as reconnecting, moving annotation points, or canceling annotation points. And improve the deletion efficiency. The logic code of the operation is as follows:

[0084]

[0085]

[0086] In a specific application scenario, according to the connection combination configuration JSON, based on the preset connection rules, calculate whether the previous and next annotation points of this annotation point already exist on the canvas. If so, connect them to each other. Among them, the connection is mutual. If point F is connected to the other three points X, Y, and Z, in addition to referencing the instances of FX, FY, and FZ lines at point F, it is also necessary to bind the line instances to the X, Y, and Z point instances through calculation. The purpose of this is to achieve that when point F is moved, it is convenient to make FX, FY, and FZ lines move with point F through calculation. The code for describing the connection operation is as follows:

[0087]

[0088]

[0089] When moving a marked point, a difficulty is how to make all the lines connected to it move with it. In this embodiment, when connecting lines, the data.name1 of the line object is assigned to the new point currently to be drawn, and the data.name2 is assigned to another point that already exists on the canvas. According to the order of the coords parameters of the drawLine() method, [x1, y1, x2, y2], it is guaranteed that name1 and [x1, y1] are matched. Then when moving a certain marked point, it is determined that among all the bound line objects of this marked point, if name1 is the currently moving point, then the x1 and y1 positions of the line are synchronized with the position to which the point moves. If name2 is the currently moving marked point, then the x2 and y2 positions of the line are synchronized with the position to which the marked point moves. In this way, the line moves with the point.

[0090] In the process of secondary annotation, the current operation record stack can be added to undo the last operation through undoStack. The action function can take values ​​of new to create a new point (and line), move to move a point (and line), and del to delete a point (and line). The location and type information of each annotation point and the color and transparency of each line can be stored in real time, which is conducive to undoing annotation and avoiding the problem of losing annotations due to some unavoidable situations.

[0091] In order to improve the annotation efficiency, shortcut key operations can be supported in the secondary manual annotation process. For example: add annotation points (left click), modify annotation points (drag points), delete annotation points (DEL), undo (right click), zoom (mouse wheel), switch annotation point type (A), show hidden labels (W), show hidden annotation traces (E), drag the image (Alt + mouse drag, MacOS system is Option key + mouse drag), adjust point radius (<, >), adjust line thickness ([,]), etc., without specific restrictions.

[0092] See also Figure 4 , Figure 4 yes Figure 2 A schematic diagram of an embodiment of Figure 1 after the marking in the embodiment is completed.

[0093] In this embodiment, part of the human body is shown on the marked picture 10. Based on the displayed part of the human body, 9 marking points are marked on it: below the neck 11, right shoulder 12, left shoulder 13, left elbow 14, right elbow 15, right wrist 16, left wrist 19, right hip 17 and left hip 18. In addition to the origin itself, the above 9 marking points are also configured with text instances to interpret each marking point, and each marking point is connected to each other based on the preset connection rules. In this embodiment, the color of each connection line is black, and its transparency is 0.67. However, in other embodiments, the color and transparency of the connection line can be customized and are not limited here.

[0094] Step S24: Traverse each annotation point and each connecting line on the annotated image to obtain the determined coordinate information, determined type information of each annotation point, and the trajectory information and connection information of each connecting line, and use the determined coordinate information, determined type information of each annotation point, and the trajectory information and connection information of each connecting line as the annotation information of the image.

[0095] After the secondary annotation is completed, the annotation points and lines on the annotated image are traversed to obtain the final coordinate information, type information, and trajectory information and connection information of each annotation point, and the coordinate information, type information, and trajectory information and connection information of each connection point are used as the annotation information of the image. In order to export the annotation information of the image. The server interface can be requested to send the annotation information of the image to the server. The server interface address can be flexibly customized.

[0096] Through the above method, the image annotation method of this embodiment first determines the number and type of points to be annotated based on the image to be annotated, and then pre-annotates the image to be annotated based on the number and type of points to be annotated and annotates the image to be annotated again, and obtains multiple annotation points, and selects the target annotation point from the multiple annotation points, and connects the target annotation point with at least one other annotated annotation point, so as to obtain the annotation information of the image based on the annotation point and the connection, thereby establishing a visual correspondence between each annotation point through the connection, thereby improving the annotation effect on the annotated image, and then improving the practicality of the annotation information when the annotation information is applied. This embodiment allows users to customize the annotation points of the image to be annotated, the rules for matching the connection between points (i.e., automatic connection), the color of the custom line, and support the pre-annotation of the image annotation model on the annotation platform through custom JSON configuration. The image annotation method of this embodiment can be used in the annotation tool to achieve UI design friendliness, smooth operation, and strong extensibility (it can be used only as a tool or plug-in, or it can be integrated into an operational annotation platform). It can greatly improve the annotation efficiency.

[0097] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of the image annotation device of the present application. The image annotation device 50 includes a determination module 51, an annotation module 52, and a connection module 53. The determination module 51 is used to determine the number and type of points to be annotated based on the image to be annotated; the annotation module 52 is used to annotate the image to be annotated based on the number and type of points to be annotated, and obtain multiple annotation points; the connection module 53 is used to select a target annotation point from the multiple annotation points, and connect the target annotation point with at least one other annotated annotation point, so as to obtain the annotation information of the image based on the annotation points and the connection.

[0098] The connection module 53 is also used to connect the target marking point with other marking points based on preset connection rules; establish a corresponding relationship between the target marking point, other marking points, and the connection between the target marking point and other marking points to obtain the target marking point, other marking points, and the combination between the target marking point and other marking points.

[0099] The connection module 53 is further used to display the connection between the target marking point and other marking points based on a preset color and a preset transparency.

[0100] The labeling module 52 is also used to identify labeling points of the image to be labeled based on the number and type of the points to be labeled through the image labeling model to obtain initial labeling information of at least one pre-labeling point; use the initial labeling information of at least one pre-labeling point to label at least one pre-labeling point on the image to be labeled; perform secondary labeling on the image to be labeled to label the remaining labeling points on the image to be labeled to obtain multiple labeling points.

[0101] The annotation module 52 is also used to identify annotation points of the image to be annotated through the image annotation model to obtain the initial coordinate information and initial type information of each pre-annotated point; configure the initial coordinate information and initial type information of each pre-annotated point to obtain the label features of each pre-annotated point; and use the label features of each pre-annotated point to mark each pre-annotated point on the image to be annotated through the image annotation model.

[0102] The marking module 52 is further used to perform secondary marking on the image to be marked, so as to mark the remaining marking points on the image to be marked and determine the type of each remaining marking point; and / or move the position of at least one pre-marked point to obtain a marking point.

[0103] The above solution can improve the annotation effect on the annotated picture by annotating the picture and connecting the annotated points, thereby improving the practicality of the annotation information.

[0104] See also Figure 6 , Figure 6 6 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps of any of the above-mentioned image annotation method embodiments. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0105] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of the above-mentioned method for labeling any of the pictures. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0106] The above solution can improve the annotation effect on the annotated picture by annotating the picture and connecting the annotated points, thereby improving the practicality of the annotation information.

[0107] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 701 that can be executed by a processor, and the program instructions 701 are used to implement the steps of any of the above-mentioned picture annotation method embodiments.

[0108] The above solution can improve the annotation effect on the annotated picture by annotating the picture and connecting the annotated points, thereby improving the practicality of the annotation information.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for labeling a picture, characterized in that: The image annotation method includes: Determine the number and type of points to be annotated based on the image to be annotated; Based on the number and type of the points to be annotated, the image to be annotated is annotated accordingly to obtain a plurality of annotated points; A target annotation point is selected from the multiple annotation points, and a line is formed between the target annotation point and at least one other annotated annotation point, so as to obtain annotation information of the image based on the annotation points and the line; wherein each annotation point and each line on the annotated image is traversed to obtain determined coordinate information, determined type information of each annotation point, and track information and connection information of each line; and the determined coordinate information, determined type information of each annotation point, and track information and connection information of each line are used as the annotation information of the image; The step of selecting a target marking point from the multiple marking points and connecting the target marking point with at least one other marked marking point comprises: Connecting the target annotation point with other annotation points based on a preset connection rule; Establishing a corresponding relationship between the target marking point, the other marking points, and the line between the target marking point and the other marking points to obtain the target marking point, the other marking points, and the combination between the target marking point and the other marking points; The connection line is bound to the corresponding annotation point so that the connection line moves along with the corresponding annotation point.

2. The image annotation method according to claim 1, characterized in that: The step of selecting a target marking point from the plurality of marking points and connecting the target marking point with at least one other marked marking point comprises: The connection line between the target marking point and the other marking points is displayed based on a preset color and a preset transparency.

3. The image annotation method according to claim 1, characterized in that: After the step of establishing a corresponding relationship between the target marking point, the other marking points, and the line between the target marking point and the other marking points to obtain the target marking point, the other marking points, and the combination between the target marking point and the other marking points, the following steps are included: The target marked point and the connecting line corresponding to the target marked point are deleted by using the combination.

4. The image annotation method according to claim 1, characterized in that: The image annotation method further includes: The location information and type information of each marked point and the color and transparency of each connecting line are stored in real time to facilitate the cancellation of marking.

5. The image annotation method according to claim 1, characterized in that: The step of annotating the image to be annotated correspondingly based on the number and type of the points to be annotated to obtain a plurality of annotated points comprises: Based on the number and type of the points to be annotated, the image to be annotated is identified by using an image annotation model to obtain initial annotation information of at least one pre-annotated point; Marking the at least one pre-marked point on the to-be-marked picture using the initial marking information of the at least one pre-marked point; The image to be annotated is annotated again to annotate the remaining annotation points on the image to be annotated, thereby obtaining the plurality of annotation points.

6. The image annotation method according to claim 5, characterized in that: The step of identifying the annotation points of the image to be annotated by using the image annotation model based on the number and type of the points to be annotated to obtain initial annotation information of at least one pre-annotated point includes: Identify the annotation points of the image to be annotated by using the image annotation model to obtain initial coordinate information and initial type information of each pre-annotated point; Configuring the initial coordinate information and the initial type information of each of the pre-marked points to obtain label features of each of the pre-marked points; The step of marking the at least one pre-marked point on the image to be marked using the initial marking information of the at least one pre-marked point comprises: The image annotation model uses the label features of the pre-annotated points to annotate the pre-annotated points on the image to be annotated.

7. The image annotation method according to claim 5, characterized in that: The step of performing secondary annotation on the image to be annotated to annotate the remaining annotation points on the image to be annotated to obtain the plurality of annotation points comprises: Performing secondary annotation on the image to be annotated, so as to mark the remaining annotation points on the image to be annotated, and determining the type of each of the remaining annotation points; and / or The position of the at least one pre-marked point is moved to obtain the marked point.

8. A device for labeling a picture, characterized in that: The image annotation device comprises: A determination module, used to determine the number and type of points to be annotated based on the image to be annotated; A labeling module, used to label the image to be labeled accordingly based on the number and type of the points to be labeled, to obtain multiple labeling points; A line connection module is used to select a target annotation point from the multiple annotation points, and connect the target annotation point with at least one other annotated annotation point to obtain the annotation information of the image based on the annotation points and the lines, wherein each annotation point and each line on the annotated image is traversed to obtain the determined coordinate information, determined type information of each annotation point, and the trajectory information and connection information of each line; the determined coordinate information, determined type information of each annotation point, and the trajectory information and connection information of each line are used as the annotation information of the image; and the line connection module is also used to connect the target annotation point with other annotation points based on preset line connection rules; establish a corresponding relationship between the target annotation point, the other annotation points, and the line between the target annotation point and the other annotation points, and obtain the target annotation point, the other annotation points, and the combination of the target annotation point and the other annotation points; wherein the line is bound to the corresponding annotation point so that the line moves with the corresponding annotation point.

9. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the image labeling method according to any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method for labeling a picture as described in any one of claims 1 to 7 is implemented.

Images