Key point marking method and device

Abstract

The present application discloses a key point annotation method and device, when the method includes: obtaining configuration data set for an image to be annotated, the configuration data includes a target tag library containing multiple annotation tags and one or more annotation areas divided according to the target tag library, wherein each annotation area includes multiple annotation tags with a sequential relationship; detecting the annotation operation performed by the user in the image to be annotated; determining the target annotation tag and the annotation position corresponding to the annotation operation, and drawing the corresponding target annotation point at the annotation position and displaying the corresponding target annotation tag; in the target annotation area where the target annotation tag is located, determining the adjacent annotation tag of the target annotation tag; if the adjacent annotation tag has a corresponding annotation point in the image to be annotated, then drawing a connecting line between the annotation point and the target annotation point. The embodiments of the present application can effectively improve the annotation efficiency of the annotator.

Description

Technical Field

[0001] The embodiments of the present application relate to data processing technology, and more particularly to a key point labeling method and device. Background Art

[0002] Key point detection refers to determining the coordinates of key locations of interest in an image or video. For example, in face key point detection, the coordinates of key locations such as the inner and outer corners of the eyes, the tip of the nose, and the corners of the mouth are determined from an image that includes the face. Key point detection is an important part of computer vision applications and plays an important role in face recognition, expression recognition, gesture recognition, and other fields.

[0003] The key point detection algorithm includes a key point detection algorithm based on deep learning. This key point detection algorithm requires a large amount of training data when training the deep learning model. These training data can be pictures with key points annotated. Therefore, how to annotate the key points of the picture is very important. Key point annotation is that the annotator finds the key points in the image according to the requirements of the algorithm personnel, and records and stores their coordinates to realize the training of the AI ​​model with image feature recognition function.

[0004] The key point marking methods used in the related art are mainly sequential marking method and index marking method. Both of these marking methods require the marking personnel to mark the key points one by one, which is inefficient. Summary of the invention

[0005] The present application provides a key point labeling method and device to solve the problem of low labeling efficiency caused by manual key point labeling in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a key point labeling method, the method comprising:

[0007] Acquire configuration data set for the image to be annotated, the configuration data comprising a target label library including a plurality of annotation labels and one or more annotation regions divided according to the target label library, wherein each annotation region comprises a plurality of annotation labels having an order relationship;

[0008] Detecting a labeling operation performed by a user in the image to be labeled;

[0009] Determine the target annotation label and the annotation position corresponding to the annotation operation, draw the corresponding target annotation point at the annotation position and display the corresponding target annotation label;

[0010] In the target marking area where the target marking label is located, determining adjacent marking labels of the target marking label;

[0011] If the adjacent annotation label has a corresponding annotation point in the image to be annotated, a connecting line between the annotation point and the target annotation point is drawn.

[0012] In a second aspect, an embodiment of the present application further provides a key point marking method, the method comprising:

[0013] Acquire configuration data set for the image to be annotated, the configuration data comprising a target label library including a plurality of annotation labels and one or more annotation regions divided according to the target label library, wherein each annotation region comprises a plurality of annotation labels having an order relationship;

[0014] Pre-annotating the image to be annotated using a preset annotation model, so as to draw a plurality of annotation points in the image to be annotated and display annotation labels corresponding to the annotation points;

[0015] Traversing each annotation point, and determining the adjacent annotation labels of the annotation label according to the annotation label of the currently traversed annotation point and the annotation area where the annotation point is located;

[0016] Draw a connecting line between the annotation point corresponding to the adjacent annotation label and the current annotation point.

[0017] In a third aspect, an embodiment of the present application further provides a key point marking device, the device comprising:

[0018] A first configuration data acquisition module is used to acquire configuration data set for the image to be annotated, wherein the configuration data includes a target label library including a plurality of annotation labels and one or more annotation areas divided according to the target label library, wherein each annotation area includes a plurality of annotation labels having an order relationship;

[0019] A labeling operation detection module, used to detect the labeling operation performed by the user in the image to be labeled;

[0020] A target annotation point drawing module is used to determine the target annotation label and the annotation position corresponding to the annotation operation, and draw the corresponding target annotation point at the annotation position and display the corresponding target annotation label;

[0021] A first adjacent annotation label determination module is used to determine adjacent annotation labels of the target annotation label in the target annotation area where the target annotation label is located;

[0022] The first connection line drawing module is used to draw a connection line between the adjacent annotation label and the target annotation point if the adjacent annotation label has a corresponding annotation point in the image to be annotated.

[0023] In a fourth aspect, an embodiment of the present application further provides a key point marking device, the device comprising:

[0024] A second configuration data acquisition module is used to acquire configuration data set for the image to be annotated, wherein the configuration data includes a target label library including a plurality of annotation labels and one or more annotation areas divided according to the target label library, wherein each annotation area includes a plurality of annotation labels having an order relationship;

[0025] A pre-annotation module, used to pre-annotate the image to be annotated using a preset annotation model, so as to draw a plurality of annotation points in the image to be annotated and display an annotation label corresponding to each annotation point;

[0026] The second adjacent annotation label determination module is used to traverse each annotation point and determine the adjacent annotation labels of the annotation label according to the annotation label of the currently traversed annotation point and the annotation area where the annotation point is located;

[0027] The second connection line drawing module is used to draw a connection line between the annotation point corresponding to the adjacent annotation label and the current annotation point.

[0028] In a fifth aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0029] one or more processors;

[0030] a storage device for storing one or more programs,

[0031] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0032] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and the program implements the above method when executed by a processor.

[0033] The technical solution provided by this application has the following beneficial effects:

[0034] In this embodiment, by setting the configuration data of the image to be annotated in advance, the target label library can be divided into one or more annotation areas, each of which includes multiple annotation labels with a sequential relationship. When the user performs an annotation operation on the image to be annotated, the target annotation point is drawn at the annotation position corresponding to the annotation operation and the corresponding target annotation label is displayed. Then, the adjacent annotation label of the target annotation label can be determined according to the target annotation area where the target annotation label is located. If the adjacent annotation label has a corresponding annotation point in the image to be annotated, a connecting line between the annotation point and the target annotation point is drawn, so that the adjacent annotation points in the same annotation area can be connected to each other, so that the annotator can clearly know the relationship between the current marked point and other points and in which area it is, thereby improving the annotation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of a key point marking method embodiment provided in Embodiment 1 of the present application;

[0036] Figure 2 This is a schematic diagram of a label library provided in Example 1 of the present application;

[0037] Figure 3 This is a visualization diagram of the left eye marking area provided in Example 1 of the present application;

[0038] Figure 4 This is a schematic diagram of marking the left eye area and the left eyebrow area provided in Example 1 of the present application;

[0039] Figure 5 It is a schematic diagram of the toolbar area of ​​the annotation page provided in the first embodiment of the present application;

[0040] Figure 6 This is a flow chart of a key point marking method embodiment provided in Embodiment 2 of the present application;

[0041] Figure 7 It is a structural block diagram of a key point marking device embodiment provided in Embodiment 3 of the present application;

[0042] Figure 8 It is a structural block diagram of a key point marking device embodiment provided in Embodiment 4 of the present application;

[0043] Fig. 9 It is a structural schematic diagram of an electronic device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0045] Embodiment 1

[0046] Figure 1 A flowchart of a key point annotation method embodiment provided in Example 1 of the present application. This embodiment can be applied to an annotation tool. Exemplarily, the annotation tool may include a Web-based annotation tool. The advantage of using a Web-based annotation tool is that the annotator does not need to install various environmental dependencies, and can implement image annotation with low threshold, high efficiency, and friendliness.

[0047] like Figure 1 As shown, the embodiment of the present application may include the following steps:

[0048] Step 110, obtaining configuration data set for the image to be annotated, wherein the configuration data includes a target label library including multiple annotation labels and one or more annotation areas divided according to the target label library, wherein each annotation area includes multiple annotation labels having an order relationship.

[0049] In one implementation, the configuration data may be data pre-configured in the annotation tool by a developer or an administrator of the annotation tool for the image to be annotated.

[0050] In one example, the configuration data may include a target tag library, which is a tag library selected by a developer or manager from multiple tag libraries, wherein each tag library may include multiple annotation tags, and the annotation tags are used to distinguish different annotation points, which can also be understood as the identification of the annotation points. It should be noted that the number of annotation tags contained in each tag library is different, and the number depends on different annotation targets. For example, assuming that the annotation task is a facial key point annotation task, depending on the different annotation targets, some annotation targets require 223 facial key points, some require 97 facial key points, and some only need to adjust some facial contour key points such as 44 facial key points.

[0051] For example, in Figure 2 In the label library schematic diagram, two label libraries are shown, namely the "face key point-223" label library and the "face key point-44" label library, wherein the "face key point-223" label library contains 223 annotation labels, and the "face key point-44" label library contains 44 annotation labels, and "0", "1", "2"... "222" and so on in the label library represent annotation labels.

[0052] In another example, the configuration data may also include one or more annotated areas divided according to the target tag library, wherein each annotated area also has a corresponding area identifier, and each annotated area may include multiple annotated tags with a sequential relationship. In practice, the multiple annotated tags with a sequential relationship included in each annotated area may be listed in the form of an enumeration, or in the form of a tag range (such as [start, end], where start represents a starting point tag and end represents an end point tag), which is not limited in this embodiment.

[0053] For example, for a face annotation task, the annotation areas may include but are not limited to: facial contour area, left eyebrow area, right eyebrow area, left eye area, right eye area, left eyeball area, right eyeball area, nose area, upper lip area, lower lip area, left nasolabial fold area, right nasolabial fold area, etc. Each annotation area has a preset range of annotation labels. For example, assuming that the target label library is "face key points-223", the ranges of the annotation labels contained in each annotation area divided for the above face annotation task are: face contour area [0,32], left eyebrow area [33,46], right eyebrow area [47,60], left eye area [61,82], right eye area [83,104], left eyeball area [105,110], right eyeball area [111,116], nose area [117,144], upper lip area [145,178], lower lip area [179,208], left nasolabial fold area [209,215], right nasolabial fold area [216,222].

[0054] In one implementation, the one or more annotation areas may be configured using JSON data, for example:

[0055]

[0056] The above JSON data configures a marked area with the area identifier "left eye", and the marked area contains 22 marking tags 61-82. In other words, the configuration data limits the use of the 22 marking tags 61-82 to mark the "left eye" area.

[0057] Step 120: Detecting the annotation operation performed by the user in the image to be annotated.

[0058] In this embodiment, the annotation operation may be an operation in which a user (annotation user or annotator) adds annotation points in the image to be annotated.

[0059] In practice, before the annotator annotates the image to be annotated, the image to be annotated needs to be imported into the annotation tool. In one embodiment, before step 120, the following steps may also be included:

[0060] Open the annotation page; obtain the size of the image to be annotated and the size of the visible area of ​​the annotation page; if the size of the image to be annotated is larger than the size of the visible area of ​​the annotation page, adjust the size of the image to be annotated according to the size of the visible area, and display the adjusted image to be annotated in the visible area.

[0061] Specifically, in this annotation tool, different annotation tasks may have different annotation pages. For example, when the annotation task is a face annotation task, the annotation user may open the face annotation page. Exemplarily, the annotation page may be a Web page.

[0062] In practice, the annotation page can be divided into several areas according to its functions, such as the toolbar area, the annotation area display area, and the visible area. Among them, the toolbar area is an area that comprehensively displays the tools that can be used during annotation, allowing users to use them conveniently. For example, the toolbar area can include various tools such as point type setting, label visualization setting, annotation trace setting, point radius setting, line thickness setting, submit, cancel, etc.

[0063] The annotation area display area is used to display the non-visual annotation areas (such as JSON data) configured by developers or managers in a visual form to the annotators in the annotation page to assist the annotators in annotating each area. For example, the annotation tags in the annotation area of ​​"left eye" in the above JSON example can be visualized in the annotation page as follows: Figure 3 When displaying each annotated area, the annotated areas can be displayed in order according to the range of the annotated labels of each annotated area. For example, in the face annotation task with the target label library of "Face Key Points-223", the visualized annotated areas are represented from top to bottom as follows: face contour area [0,32], left eyebrow area [33,46], right eyebrow area [47,60], left eye area [61,82], right eye area [83,104], left eyeball area [105,110], right eyeball area [111,116], nose area [117,144], upper lip area [145,178], lower lip area [179,208], left nasolabial fold area [209,215], right nasolabial fold area [216,222].

[0064] The visible area is used to display the image to be annotated. However, since the visible area is limited, in order to see the full image of the image to be annotated in the visible area during initialization, the size of the image to be annotated can be compared with the size of the visible area before displaying the image to be annotated. If the size of the image to be annotated is less than or equal to the size of the visible area, the image to be annotated can be directly displayed in the visible area. If the size of the image to be annotated is larger than the size of the visible area, the size of the image to be annotated needs to be adjusted so that the size of the image to be annotated matches the size of the visible area, and the adjusted image to be annotated is displayed in the visible area.

[0065] In one implementation, adjusting the size of the image to be annotated may include the following process:

[0066] The aspect ratio of the image to be annotated (i.e., the ratio of the image height to the width, which may be referred to as the first aspect ratio for ease of description) and the aspect ratio of the visible area (which may be referred to as the second aspect ratio) are obtained. If the first aspect ratio is greater than or equal to the second aspect ratio, it indicates that the image to be annotated is relatively tall. In this case, the height of the image to be annotated may be adjusted to the height of the visible area, and the width of the image to be annotated may be scaled proportionally. The scaling may include: calculating the aspect ratio of the image to be annotated, and adjusting the width of the image to be annotated to the product of the height of the visible area and the aspect ratio.

[0067] If the first aspect ratio is smaller than the second aspect ratio, it means that the image to be annotated is relatively wide. At this time, the width of the image to be annotated can be adjusted to the width of the visible area, and the height of the image to be annotated can be scaled proportionally, wherein the proportional scaling includes: calculating the aspect ratio of the image to be annotated, and adjusting the height of the image to be annotated to the product of the width of the visible area and the above aspect ratio.

[0068] After the image to be annotated is displayed in the visible area, the annotator can perform an annotation operation on the image to be annotated. It should be noted that when the annotator performs the annotation operation, the marking of each point is determined by the rules given by the task creator. For example, Figure 3 In the example, the administrator of the current task stipulates that point 72 must be marked at the corner of the eye.

[0069] In practice, since there may be many points to be annotated in the image to be annotated, in order to avoid the front-end performance bottleneck (page freezes due to a large number of DOMs), a canvas can be used for drawing and interaction in the annotation page. The annotator can perform annotation operations on the image to be annotated on the Canvas, for example, the annotator can click the Canvas with the left mouse button.

[0070] Step 130 : determining a target annotation label and an annotation position corresponding to the annotation operation, and drawing a corresponding target annotation point at the annotation position and displaying a corresponding target annotation label.

[0071] During implementation, the click position of the user in the Canvas may be used as the marking position, and this embodiment does not limit the method of obtaining the click position.

[0072] As for the method of determining the target annotation label corresponding to the annotation operation, the user can manually select the target annotation label, or the target annotation label can be determined by automatic calculation. For example, the following methods may be included but are not limited to. In practice, one of the methods may be selected to determine the target annotation label according to the situation:

[0073] Method 1: Use the annotation tag selected by the user from the annotation area display area as the target annotation tag corresponding to the annotation operation. In this method, before or after performing the annotation operation, the annotator can select the target annotation area in the annotation area display area of ​​the annotation page, and then select a annotation tag from the target annotation area. This method can use the annotation tag selected by the user as the target annotation tag. For example, Figure 3 In the example, assuming that the user is annotating the left eye area, the user can select the "left eye" annotation area. If the user clicks on the annotation label "61" and then clicks to annotate in the image to be annotated, the annotation label "61" can be used as the target annotation label corresponding to the current annotation operation.

[0074] Method 2: If the labeling operation is the first labeling operation in the current target labeling area, the first label in the target labeling area is used as the target labeling label. This method is to automatically determine the target labeling label. For example, Figure 3 In the example, assuming that the user is annotating the left eye area and the current annotation is the first annotation of the area, the first annotation label "61" of the area can be automatically used as the target annotation label.

[0075] Method three, if the marking operation is not the first marking operation in the current target marking area, then according to the marking label corresponding to the previous marking operation, the next unoccupied marking label in the target marking area is selected as the target marking label; if there is no next unoccupied marking label but there is an unoccupied marking label before the marking label corresponding to the previous marking operation, the target marking label is determined by taking the modulus method.

[0076] The third method is also a method for automatically determining the target annotation label. In this method, if the current annotation operation is not the first annotation operation in the current annotation area, the next unoccupied annotation label in the current annotation area can be automatically selected as the target annotation label according to the annotation label corresponding to the previous annotation operation (i.e., the previous annotation label). Figure 3 In the example, assuming that the last annotation label is "72", if "73" to "82" are not occupied, the annotation label "73" can be used as the target annotation label; if "73" to "75" are occupied and "76" to "82" are not occupied, the annotation label "76" can be used as the target annotation label. The occupied label means that there are already annotated points in the image to be annotated. In practice, in order to distinguish whether the annotation label is occupied, the occupied annotation label can be grayed out in the visualized annotation area.

[0077] However, if all the annotation tags after the previous annotation tag in the current annotation area are occupied (that is, there is no next unoccupied annotation tag), but there is an unoccupied annotation tag before the previous annotation tag, then the target annotation tag can be selected from the unoccupied annotation tags by taking the modulo method. The modulo formula can be: remainingPoints[(index+1)%remainPoints.length], where remainingPoints is the unoccupied annotation tag, remainingPoints.length is the number of unoccupied annotation tags, and index is the previous annotation tag. For example, Figure 3 In the example, assume that the last annotation label is "72". If "73" to "82" are all occupied and "66" to "71" are not occupied, then the index is 72, remainingPoints.length is 6, and the modulo result is 73%6=1, that is, the first annotation label "66" is taken as the target annotation label from the unoccupied annotation labels.

[0078] After determining the target annotation label and the annotation position corresponding to the current annotation operation, the corresponding annotation point can be drawn in Canvas, and the target annotation label corresponding to the annotation point can be indicated.

[0079] In one embodiment, the configuration data may further include: a designated annotation tag for indicating a designated position, which may exemplarily include an inflection point position or other special position indicating a special part. In implementation, JSON data may be used to configure the designated annotation tag, for example, a set or array of designated annotation tags may be described by JSON data. In the case of the presence of the designated annotation tag, the step of drawing the corresponding target annotation point at the annotation position in step 130 may further include the following steps:

[0080] If the target annotation tag is not the designated annotation tag, a corresponding target annotation point is drawn at the annotation position according to a first preset tag attribute; if the target annotation tag is the designated annotation tag, a corresponding target annotation point is drawn at the annotation position according to a second preset tag attribute.

[0081] Specifically, after determining the target annotation tag corresponding to the current annotation operation, it can be determined whether the target annotation tag exists in the set of previously configured designated annotation tags. If not, it means that the target annotation tag is not the designated annotation tag, and the corresponding target annotation point can be drawn at the annotation position according to the first preset tag attribute. If the target annotation tag exists in the set of designated annotation tags, it means that the target annotation tag is the designated annotation tag, and the corresponding target annotation point can be drawn at the annotation position according to the second preset tag attribute.

[0082] It should be noted that the first preset label attribute and the second preset label attribute are set to distinguish whether a certain annotation point is an inflection point or other special point. For example, the first preset label attribute and the second preset label attribute may both include one or a combination of the shape, color, transparency, etc. of the annotation point. The first preset label attribute and the second preset label attribute may differ in one or a combination of the shape, color, transparency, etc. For example, the color of the inflection point or other special point is set to be different from the color of the ordinary annotation point. In other implementations, the second preset label attribute may also be used as a supplement to the first preset label attribute, for example, based on the first preset label attribute, the inflection point or other special point is highlighted.

[0083] By configuring the specified annotation tags, you can improve the annotation efficiency. When annotating, the annotator can first annotate the annotation points corresponding to the specified annotation tags, and then annotate other annotation points in groups to better improve the annotation efficiency.

[0084] When displaying the target annotation label, it can be displayed in a preset area of ​​the target annotation point. For example, the target annotation label can be displayed near the target annotation point. In order to make the annotation page neater, the target annotation label can also be superimposed on the target annotation point. For example, assuming that the target annotation point is represented by a square (such as a rectangle or a square, of course, it is not limited to a square, and other shapes such as a solid circle with a preset radius are all possible), a target annotation label of a preset size can be superimposed in the square, such as Figure 4 The annotation points and annotation labels in the left eye area and left eyebrow area annotation schematic diagram are shown.

[0085] Step 140 : determining adjacent annotation tags of the target annotation tag in the target annotation region where the target annotation tag is located.

[0086] In this step, the adjacent annotation tags of the target annotation tag may include the previous annotation tag of the target annotation tag and / or the next annotation tag of the target annotation tag. In the annotation area, the two endpoint annotation tags each have one adjacent annotation tag, and the other non-endpoint annotation tags each have two adjacent annotation tags before and after. For example, Figure 3 In the figure, the endpoint annotation label "61" has an adjacent annotation label "62", the endpoint annotation label "82" has an adjacent annotation label "81", and other non-endpoint annotation labels, such as the annotation label "62", have adjacent annotation labels including the previous annotation label "61" and the next annotation label "63".

[0087] Step 150: If the adjacent annotation label has a corresponding annotation point in the image to be annotated, a connecting line is drawn between the annotation point and the target annotation point.

[0088] In this embodiment, a line can be automatically drawn between two adjacent annotation points in the same annotation area. Specifically, after obtaining the adjacent annotation label of the current target annotation label, it can be determined whether the adjacent annotation label exists in the annotation labels of all the annotated annotation points in the image to be annotated. If it exists, it means that the adjacent annotation label has a corresponding annotation point in the image to be annotated. Then, a connecting line can be drawn in the Canvas between the annotation point of the adjacent annotation label and the current target annotation point. For example, Figure 4 In , if the current target annotation point is the point with the target annotation label "62" (for the convenience of description, it can be called point "62"), if point "61" and point "63" have been drawn in the image to be annotated, then the connecting line between point "62" and point "61" can be drawn, and the connecting line between point "62" and point "63" can be drawn. Assuming that only point "61" is drawn in the image to be annotated, and point "63" has not been drawn, then after drawing point "62", only the connecting line between point "62" and point "61" is drawn, so as to realize the connection between adjacent annotation points, such as Figure 4 shown.

[0089] In one embodiment, the configuration data may further include connection line attributes set for each marked area. For example, the connection line attributes may include the shape (such as a straight line, a wavy line, etc.) and color of the connection line. Then, step 150 may further include the following steps:

[0090] A connecting line between the marking point and the target marking point is drawn according to the connecting line attribute corresponding to the marking area where the target marking label is located.

[0091] In this embodiment, when drawing a connection line, it can be drawn according to the connection line attributes of the current marked area. Configuring the connection line attributes of each area can facilitate distinguishing different marked areas. Figure 4 In the example, if the connection line attribute of the "Left Eyebrow" area is set to a green color, and if the connection line attribute of the "Left Eye" area is set to a blue color, different annotation areas can be distinguished according to the connection line colors.

[0092] In other implementations, the connection line attributes can also be configured as attribute variables, which can be displayed in the toolbar area of ​​the annotation page or in the blank position of the corresponding annotation area when being visualized. In this way, the annotator can customize the color and / or shape of the connection lines of different annotation areas according to the visualized connection line attributes.

[0093] In one embodiment, the configuration data may also include region visibility settings for each annotation region, and label visibility settings for the annotation labels. The region visibility settings are used to hide or display the annotation points and the corresponding annotation labels; the label visibility settings are used to hide or display the corresponding annotation labels, but the annotation points are still visible. This embodiment may also include the following steps:

[0094] The area visibility settings of each annotation area and the label visibility settings are visualized in the annotation page.

[0095] In one example, the area visibility setting and the label visibility setting may be visualized in the toolbar area of ​​the annotation page, or may be displayed in a blank position of each annotation area, which is not limited in this embodiment.

[0096] For example, the area visibility setting is used to visualize the blank space in the annotation area. Figure 3 As shown in Part A of . For another example, the label visibility setting in the toolbar area after the visualization mark can be referred to Figure 5 As shown in part B.

[0097] After visualizing the area visibility setting, this embodiment may further include the following steps:

[0098] When it is detected that the user triggers the visibility setting for a certain annotation area, all annotation labels and corresponding annotation points of the annotation area are hidden in the image to be annotated.

[0099] In this embodiment, the annotator can hide or display the entire annotated area. Figure 3 In the example, when the annotator selects the "left eye" annotation area (assuming that no specific annotation label is selected in the annotation area at this time), the annotator clicks Figure 3 The mark of part A in the image (after clicking, the mark of part A will become a preset hidden mark, and after clicking the hidden mark, the selected area or annotation point will change from a hidden state to a real state), triggers the hiding operation of the entire "left eye" annotation area. After detecting the hiding operation, this annotation tool can hide the annotation labels "61" to "82" and the corresponding annotation points in the image to be annotated.

[0100] After visualizing the area visibility setting, this embodiment may further include the following steps:

[0101] When it is detected that the user triggers the visibility setting of a certain annotation area for one or more annotation tags in the annotation area, the one or more annotation tags and the corresponding annotation points are hidden in the image to be annotated.

[0102] In this embodiment, the annotator can hide or display some of the annotated points in the annotated area. Figure 3 In the example, when the annotator selects the two annotation labels "61" and "72" in the "Left Eye" annotation area (for example, you can hold down the ctrl key to select multiple annotation labels at the same time), the annotator clicks Figure 3 The mark of part A in the figure triggers the hiding operation of the two annotation points "61" and "72" in the "left eye" annotation area. After detecting the hiding operation, the annotation tool can hide the annotation labels "61" and "72" and the corresponding annotation points in the image to be annotated.

[0103] After visualizing the tag visibility setting, this embodiment may further include the following steps:

[0104] When it is detected that the user triggers the tag visibility setting for a certain annotation area or one or more annotation tags, the annotation tags of the annotation area or the annotation points corresponding to the one or more annotation tags are hidden in the image to be annotated.

[0105] For example, in Figure 3 In the example, when the annotator selects the "left eye" annotation area (assuming that no specific annotation label is selected in the annotation area at this time), the annotator clicks Figure 5 , the label "61" to "82" in the "left eye" annotation area will be hidden. After detecting the hiding operation, the annotation tool can hide the annotation labels "61" to "82" in the image to be annotated, but the corresponding annotation points will be displayed. For another example, when the annotator selects the two annotation labels "61" and "72" in the "left eye" annotation area, and then clicks Figure 5 The mark of part B in the figure triggers the hiding operation of the annotation labels of the two annotation points "61" and "72" in the "left eye" annotation area. After detecting the hiding operation, the annotation tool can hide the annotation labels "61" and "72" in the image to be annotated and only display the corresponding annotation points.

[0106] In one embodiment, the configuration data may further include a marking point type setting, and this embodiment may further include the following steps:

[0107] The annotation point type setting is visualized in the annotation page; the annotation point type set by the user for the target annotation point is detected and the annotation point type is recorded; and the target annotation point is marked according to the annotation point type.

[0108] Exemplarily, the annotation point type may include but is not limited to the following types: visible, invisible, and occluded. The visible type means that the current annotation point is completely visible on the canvas, without occlusion, and does not exceed the image boundary; the occluded type means that part or all of the current annotation point is covered by an occluder; the invisible type means that the current annotation point is an annotation point that exceeds the image boundary. The annotation point type of each annotation point is the visible type by default.

[0109] If a mark point is obscured when it is being marked, for example, when marking the left eyebrow, if part of the left eyebrow is obscured by hair, in order to form a complete left eyebrow marking result and improve the training accuracy of the AI ​​model, the obscured area will also be outlined. The annotator can set the mark point type of the obscured mark point to the obscured type.

[0110] If, when annotating a mark point, the mark point exceeds the boundary of the image, for example, when annotating the facial contour of an image that only shows part of the face (for example, only half of the facial contour is shown in the image), in order to form a complete facial contour annotation result and improve the training accuracy of the AI ​​model, the other part of the facial contour that is not shown in the image will also be outlined. In this case, the annotator can set the mark point type of the mark point of the other part of the facial contour that exceeds the image boundary to an invisible type.

[0111] When recording the annotation point type, the visible type may be recorded as a value of 2, the occluded type may be recorded as a value of 1, and the invisible type may be recorded as a value of 0.

[0112] After determining the type of the annotation point, the type can also be marked in the annotation point to distinguish different types of annotation points. For example, different colors can be used to distinguish different types of annotation points. If the annotation point type is a visible type, the color of the annotation point can be set to blue; if the annotation point type is an obscured type, the color of the annotation point can be set to purple; if the annotation point type is an invisible type, the color of the annotation point can be set to red.

[0113] In order to improve the annotation efficiency of the annotators, corresponding shortcut keys can be set for various operations. Figure 5In the , the shortcut key for setting label visibility is the "W" key, and the shortcut key for switching point types is the "A" key (when implemented, pointType = (pointType + 1)% 3 can be used to control the switching of control point types). Other shortcut keys provided by this annotation tool include but are not limited to: adding points (left click), modifying points (dragging points), deleting points (DEL), undoing (right click), zooming (mouse wheel), switching point types (A), showing and hiding annotation traces (the E key controls the display or hiding of all annotation traces, which is convenient for comparison with the original image), dragging images (Alt + mouse drag, MacOS system is Option key + mouse drag), adjusting point radius (<,>), adjusting line thickness ([,]), etc.

[0114] In one embodiment, after step 150, the present embodiment may further include the following steps:

[0115] Create a connection line instance of the connection line; associate and store the connection line instance, the marking position of the target marking point, and the target marking label; obtain the marking position of the marking point corresponding to the adjacent marking label; associate and store the adjacent marking label, the marking position of the marking point corresponding to the adjacent marking label, and the connection line instance.

[0116] In this embodiment, in order to realize the binding of the annotation points and the connecting lines, and to facilitate the subsequent interaction of the annotation points, the corresponding connecting lines will have corresponding interactive linkages, and a connecting line instance of the connecting line can be created according to the sequence of two adjacent annotation points connected by the connecting line. Exemplarily, the connecting line instance can include a starting point position and an end point position, wherein the starting point position can be represented by the horizontal coordinate and vertical coordinate of the starting point in the canvas, and the end point position can be represented by the horizontal coordinate and vertical coordinate of the end point in the canvas. That is, the connecting line instance can be represented as [x1, y1, x2, y2], where x1 and y1 are the horizontal coordinate and vertical coordinate of the starting point of the connecting line, and x2 and y2 are the horizontal coordinate and vertical coordinate of the end point of the connecting line.

[0117] In one example, if the adjacent annotation tag is the previous tag of the current target annotation tag, the annotation position of the annotation point corresponding to the adjacent annotation tag is used as the starting position, and the annotation position of the target annotation point is used as the end position to establish a connecting line instance, that is, line1 = makeLine([prev.data.x, prev.data.y, x, y]), where prev.data.x and prev.data.y are the horizontal and vertical coordinates of the previous annotation point, respectively, and x and y are the horizontal and vertical coordinates of the current target annotation point, respectively. For example, in Figure 3In the figure, for the connecting line between point "62" and point "61", since point "61" is the previous label of the current point "62", the connecting line instance of the connecting line can be expressed as (the horizontal coordinate of point "61", the vertical coordinate of point "61", the horizontal coordinate of point "62", the vertical coordinate of point "62").

[0118] In another example, if the adjacent annotation label is the next label of the target annotation label, the annotation position of the target annotation point is used as the starting position, and the annotation position of the annotation point corresponding to the adjacent annotation label is used as the end position to establish the connecting line instance of the connecting line, that is, line2 = makeLine([x, y, next.data.x, next.data.y]), where x and y are the horizontal and vertical coordinates of the current target annotation point, and next.data.x and next.data.y are the horizontal and vertical coordinates of the next annotation point, respectively. For example, in Figure 3 In the figure, for the connecting line between point "62" and point "63", since point "63" is the next label of the current point "62", the connecting line instance of this connecting line can be expressed as (the horizontal coordinate of point "62", the vertical coordinate of point "62", the horizontal coordinate of point "63", the vertical coordinate of point "63").

[0119] After the connection line instance is established, the annotation point, connection line instance and annotation label can be combined into a "group". The "group" is a whole, so that some operations can be performed on this whole, such as defining some properties for the group or associating some object instances, scaling, deleting, etc. Specifically, the connection line instance, the annotation position of the current target annotation point and the target annotation label can be stored in association. At the same time, the annotation position of the annotation point corresponding to the adjacent annotation label can be obtained, and the adjacent annotation label, the annotation position of the annotation point corresponding to the adjacent annotation label and the connection line instance can be stored in association.

[0120] When implementing, the connecting line instance, the marking position and the target marking label of the target marking point can be stored in the data field of the target marking point. In one example, the connecting line instance of the target marking point can be recorded as [line1, line2], where prev.data.line2 = line1, that is, line1 of the current marking point is line2 of the previous marking point. If the previous marking point is an endpoint, prev.data.line1 is null. For example, in Figure 3In , the current annotation point is point "62", the previous annotation point is point "61", prev is the object of point "61", and point "61" is the endpoint, which only needs to bind the connecting line (61-62), so prev.data.line1 is actually a null value, and prev.data.line2 is (61-62). In addition, next.data.line1 = line2, that is, line2 of the current annotation point is line1 of the next annotation point. For example, in Figure 3 In the example, the current marked point is point "62", and the next marked point is point "63" which has been marked. Next is the object of point "63", so next.data.line1 is assigned the value of line2.

[0121] In one embodiment, the annotator can move and adjust the position of the annotated points in the image to be annotated. When the annotated points are moved, the corresponding connecting lines will also change. Based on this, the embodiment may further include the following steps:

[0122] When it is detected that the target annotation point is moved, the annotation position of the target annotation point after the movement is determined; according to the annotation position after the movement, the connection line instance is updated; and the corresponding connection line is redrawn based on the updated connection line instance.

[0123] Specifically, when the target annotation point is moved, the annotation position of the target annotation point after the move can be obtained. For line1 of the target annotation point, the end position of line1 can be updated to the annotation position after the move; for line2 of the target annotation point, the starting position of line2 can be updated to the annotation position after the move; if line1 or line2 is a null value, no update is required. After line1 and line2 of the target annotation point are updated, the updated line1 can be used as the line2 of the previous annotation point, and the updated line2 can be used as the line1 of the next annotation point to update the connection line instances of the previous and next annotation points. Then, the connection lines between the target annotation point and the previous and next annotation points can be redrawn according to the updated connection line instances, so that when the annotation point is moved, one or two connection lines of the annotation point will move with the moved position of the annotation point.

[0124] In one embodiment, the annotator may also delete the annotated points in the image to be annotated. When the annotated points are deleted, the corresponding connecting lines will also be deleted. Based on this, the embodiment may further include the following steps:

[0125] When it is detected that the target annotation point is deleted, the connection line instance is deleted, and the corresponding connection line is deleted.

[0126] Specifically, if the target annotation point is deleted, the connecting line instances line1 and line2 of the target annotation point can be deleted, as well as the line2 of the previous annotation point (if any) and the line1 of the next annotation point (if any) of the target annotation point. At the same time, the connecting lines between the target annotation point and the previous and next annotation points on the canvas need to be deleted. If they are not deleted, the subsequent interactions such as reconnecting, moving points, and undoing will have errors.

[0127] In one embodiment, the following steps may also be included:

[0128] Various operations performed by the user in the image to be annotated are captured, and operation objects are generated according to the operations; and the operation objects are stored in a preset undo stack.

[0129] Specifically, in this embodiment, an undo stack undoStack can be set, and various operations performed by the user in the image to be annotated (such as creating a new annotation point (new), moving an annotation point (move), deleting an annotation point (del), etc.) can be recorded in the undoStack to achieve the function of undoing the last operation. If the operation is move, the undoStack can also record the annotation position before the annotation point is moved and the annotation position after the movement.

[0130] In one embodiment, the following steps may also be included:

[0131] When a submission instruction triggered by a user is detected, the annotation result of the user on the image to be annotated is obtained; and the annotation result is uploaded to a server.

[0132] In this embodiment, when the annotator completes the annotation of the image to be annotated, he can click the submit button in the annotation page to upload the annotation result. For this annotation tool, when the user-triggered submission instruction is detected, it is determined that the user has completed the annotation of the image to be annotated. At this time, the annotation data of the user on the canvas for the image to be annotated can be obtained. Exemplarily, the annotation data may include but is not limited to: the annotation position of each annotation point, and the point type of each annotation point (invisible, visible, blocked, etc.). If the size of the image is adjusted when it is displayed, the annotation position can be restored according to the adjustment ratio, and the annotation result can be generated according to the restored annotation position of the annotation point and the annotation point type of the annotation point. Then, this annotation tool can request the server interface to send the annotation result to the server, and the server only needs to store the annotation result, thereby reducing the development pressure of the server. It should be noted that the server interface address can be flexibly customized, and this embodiment does not limit this.

[0133] In one implementation, the method for restoring the annotation position according to the adjustment ratio can be: determine the adjustment ratio ratio as the ratio of the original width of the image to be annotated to the adjusted width of the image to be annotated, assuming that the annotation position is (x, y), then the restored annotation position is (x*ratio, y*ratio).

[0134] In other embodiments, the annotation results may also be cached locally, for example, using the browser's localStorage to locally store the annotation results to avoid loss of annotation data due to certain circumstances.

[0135] In this embodiment, by setting the configuration data of the image to be annotated in advance, the target label library can be divided into one or more annotation areas, each of which includes multiple annotation labels with a sequential relationship. When the user performs an annotation operation on the image to be annotated, the target annotation point is drawn at the annotation position corresponding to the annotation operation and the corresponding target annotation label is displayed. Then, the adjacent annotation label of the target annotation label can be determined according to the target annotation area where the target annotation label is located. If the adjacent annotation label has a corresponding annotation point in the image to be annotated, a connecting line between the annotation point and the target annotation point is drawn, so that the adjacent annotation points in the same annotation area can be connected to each other, so that the annotator can clearly know the relationship between the current marked point and other points and in which area it is, thereby improving the annotation efficiency.

[0136] In addition, this embodiment also provides other configurations to improve the annotation efficiency, such as special display of special points (such as inflection points, etc.), providing shortcut key support for various operations, distinguishing the annotation point types of different annotation points, displaying the annotation points of different annotation point types differently, hiding all or part of the annotation points in certain annotation areas, customizing the connection line properties, undoable operations, etc. The above annotation functions are realized by customizing the configuration data, which improves the annotation efficiency and has strong extensibility (can be used only as a tool or as a plug-in), and can cover the annotation needs of various enterprise-level key points.

[0137] Embodiment 2

[0138] Figure 6 This is a flowchart of a key point labeling method embodiment provided in Embodiment 2 of the present application. The difference between this embodiment and Embodiment 1 is that this embodiment supports the AI ​​model pre-labeling function and may include the following steps:

[0139] Step 610, obtaining configuration data set for the image to be annotated, wherein the configuration data includes a target tag library including multiple annotation tags and one or more annotation areas divided according to the target tag library, wherein each annotation area includes multiple annotation tags having an order relationship.

[0140] For the description of step 610, reference may be made to step 110 in the first embodiment, which will not be repeated here.

[0141] Step 620 : pre-annotate the image to be annotated using a preset annotation model, so as to draw a plurality of annotation points in the image to be annotated and display an annotation label corresponding to each annotation point.

[0142] In this embodiment, an AI model interface can be provided to call the AI ​​model to assist in labeling. With AI assistance, the labeling personnel only need to fine-tune some points, thereby greatly improving the labeling efficiency.

[0143] Among them, the annotation model can be a model trained offline, or a model obtained through reinforcement learning or transfer learning. This embodiment does not limit the training process and training method of the annotation model.

[0144] The pre-annotation results output by the AI ​​model after pre-annotation may include but are not limited to: the annotation label of the annotation point (i.e., the pre-annotation point), the annotation coordinates, and the annotation point type, wherein the annotation coordinates may include the horizontal coordinate of the current annotation point relative to the left boundary and the vertical coordinate relative to the upper boundary.

[0145] During implementation, you can define a JSON to store the pre-annotation results. For example, the JSON data of the pre-annotation results is as follows:

[0146] {

[0147] 0:[456.12207,915.48486,2],

[0148] 1:[445.7528,991.6262,2],

[0149] 2:[439.641,1065.5913,2],

[0150] …

[0151] }

[0152] The outer layer of this JSON configuration is an object, the inner key name is the annotation tag of the annotation point, and the key value is an array of length 3. The values ​​in this array represent the horizontal coordinate relative to the left boundary, the vertical coordinate relative to the upper boundary, and the annotation point type (0 for invisible, 1 for obscured, and 2 for visible) in order.

[0153] It should be noted that in this embodiment, the pre-labeling result and the final labeling result can share a field labelFeature. Before the formal labeling, it is only necessary to recognize the image once through the AI ​​model, fill the pre-labeling result (such as the JSON above) into the LabelFeature field, and after the formal labeling is modified and submitted, the final labeling result will replace the previously pre-labeled labelFeature. Of course, it is not necessary to share a field, and it can also be divided into two fields to distinguish the pre-labeling result from the final labeling result. This embodiment does not limit this.

[0154] After the pre-marking results are obtained, the corresponding marking points are drawn on the canvas according to the marking coordinates and the marking point types, and the marking labels corresponding to the marking points are displayed.

[0155] Step 630 , traverse each annotation point, and determine the adjacent annotation tags of the annotation tag according to the annotation tag of the currently traversed annotation point and the annotation area where the annotation point is located.

[0156] Step 640: Draw a connecting line between the annotation point corresponding to the adjacent annotation label and the current annotation point.

[0157] Regarding the method of drawing the connection lines in step 630 and step 640, reference may be made to the description of step 140 and step 150 in the first embodiment, which will not be repeated here.

[0158] In this embodiment, regarding the processes of drawing connecting lines according to the connecting line attributes in the configuration data, hiding annotation points and / or annotation labels according to the area visibility settings and label visibility settings in the configuration data, switching annotation point types, moving, adjusting or deleting annotation points by creating connecting line instances and binding connecting line instances, undoing the last operation through the undo stack, generating annotation results, etc., you can refer to the relevant description of Example 1, which will not be repeated here.

[0159] Embodiment 3

[0160] Figure 7 This is a structural block diagram of an embodiment of a key point marking device provided in Embodiment 3 of the present application. The device can be applied to a marking tool and may include the following modules:

[0161] A first configuration data acquisition module 710 is used to acquire configuration data set for the image to be annotated, wherein the configuration data includes a target label library including a plurality of annotation labels and one or more annotation regions divided according to the target label library, wherein each annotation region includes a plurality of annotation labels having an order relationship;

[0162] A labeling operation detection module 720, used to detect a labeling operation performed by a user in the image to be labeled;

[0163] The target annotation point drawing module 730 is used to determine the target annotation label and the annotation position corresponding to the annotation operation, and draw the corresponding target annotation point at the annotation position and display the corresponding target annotation label;

[0164] A first adjacent annotation label determination module 740 is used to determine adjacent annotation labels of the target annotation label in the target annotation area where the target annotation label is located;

[0165] The first connection line drawing module 750 is configured to draw a connection line between the corresponding annotation point and the target annotation point if the adjacent annotation label has a corresponding annotation point in the image to be annotated.

[0166] In one embodiment, the device further comprises:

[0167] A connection line instance creation module, used to create a connection line instance of the connection line;

[0168] An associated storage module is used to associate and store the connecting line instance, the marking position of the target marking point and the target marking label; and obtain the marking position of the marking point corresponding to the adjacent marking label, and associate and store the adjacent marking label, the marking position of the marking point corresponding to the adjacent marking label and the connecting line instance.

[0169] In one embodiment, the connection line instance includes a starting point position and an end point position, and the connection line instance creation module is specifically used to:

[0170] If the adjacent annotation tag is the previous tag of the target annotation tag, the annotation position of the annotation point corresponding to the adjacent annotation tag is used as the starting position, and the annotation position of the target annotation point is used as the end position to establish a connecting line instance of the connecting line;

[0171] and / or,

[0172] If the adjacent annotation label is the next label of the target annotation label, the annotation position of the target annotation point is used as the starting position, and the annotation position of the annotation point corresponding to the adjacent annotation label is used as the end position to establish a connecting line instance of the connecting line.

[0173] In one embodiment, the device further comprises:

[0174] The marking point moving module is used to determine the marking position of the target marking point after it is detected that the target marking point is moved; update the connecting line instance according to the marking position after the movement; and redraw the corresponding connecting line based on the updated connecting line instance.

[0175] In one embodiment, the device further comprises:

[0176] The annotation point deletion processing module is used to delete the connection line instance and the corresponding connection line when it is detected that the target annotation point is deleted.

[0177] In one embodiment, the device further comprises:

[0178] The undo processing module is used to capture various operations performed by the user in the image to be annotated, generate operation objects according to the operations, and store the operation objects in a preset undo stack.

[0179] In one embodiment, the configuration data further includes: a designated annotation tag for indicating a designated location, and the target annotation point drawing module 730 is specifically used to:

[0180] If the target annotation label is not the designated annotation label, drawing a corresponding target annotation point at the annotation position according to a first preset label attribute;

[0181] If the target annotation tag is the designated annotation tag, a corresponding target annotation point is drawn at the annotation position according to a second preset tag attribute.

[0182] In one embodiment, the configuration data further includes connection line attributes set for each marked area; the first connection line drawing module 750 is specifically used to:

[0183] A connecting line between the marking point and the target marking point is drawn according to the connecting line attribute corresponding to the marking area where the target marking label is located.

[0184] In one embodiment, the configuration data further includes area visibility settings for each annotated area, and label visibility settings for the annotated labels; the apparatus further includes:

[0185] A first visualization module, used to visualize the area visibility setting of each annotation area and the label visibility setting in the annotation page;

[0186] A first region visibility triggering module is configured to hide all annotation labels and corresponding annotation points of a certain annotation region in the image to be annotated when it is detected that a user triggers the region visibility setting for the annotation region;

[0187] A second region visibility triggering module is configured to hide the one or more annotation tags and corresponding annotation points in the image to be annotated when it is detected that a user triggers the region visibility setting of the annotation region for one or more annotation tags in the annotation region;

[0188] The label visibility trigger module is used to hide the label labels of the label area or the label points corresponding to the one or more label labels in the image to be annotated when it is detected that the user triggers the label visibility setting for a certain label area or one or more label labels.

[0189] In one embodiment, the configuration data further includes a marking point type setting, and the device further includes:

[0190] A second visualization module, used for visualizing the annotation point type setting in the annotation page;

[0191] A marking point type detection module, used to detect the marking point type set by the user for the target marking point, and record the marking point type;

[0192] A marking point type marking module is used to mark the target marking point according to the marking point type.

[0193] In one embodiment, the device further comprises:

[0194] A marking page opening module, used to open a marking page, wherein the marking page includes a visible area for displaying the image to be marked;

[0195] The image adjustment module is used to obtain the size of the image to be annotated and the size of the visible area of ​​the annotation page; if the size of the image to be annotated is larger than the size of the visible area of ​​the annotation page, the size of the image to be annotated is adjusted according to the size of the visible area, and the adjusted image to be annotated is displayed in the visible area.

[0196] In one embodiment, the annotation page further includes an annotation area display region for visually displaying each annotation area according to the configuration data, and the target annotation point drawing module 730 is further used to:

[0197] Using the annotation tag selected by the user from the annotation area display area as the target annotation tag corresponding to the annotation operation;

[0198] or,

[0199] If the marking operation is the first marking operation in the current target marking area, the first marking label in the target marking area is used as the target marking label;

[0200] If the marking operation is not the first marking operation in the current target marking area, the next unoccupied marking tag in the target marking area is selected as the target marking tag according to the marking tag corresponding to the previous marking operation; if there is no next unoccupied marking tag but there is an unoccupied marking tag before the marking tag corresponding to the previous marking operation, the target marking tag is determined by taking the modulus.

[0201] In one embodiment, the device further comprises:

[0202] The annotation result acquisition module is used to acquire the annotation result of the image to be annotated by the user when a submission instruction triggered by the user is detected;

[0203] The annotation result uploading module is used to upload the annotation result to the server.

[0204] A key point marking device provided in an embodiment of the present application can execute a key point marking method in the first embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0205] Embodiment 4

[0206] Figure 8 This is a structural block diagram of an embodiment of a key point marking device provided in Embodiment 4 of the present application. The device can be applied to a marking tool and may include the following modules:

[0207] A second configuration data acquisition module 810 is used to acquire configuration data set for the image to be annotated, wherein the configuration data includes a target label library including a plurality of annotation labels and one or more annotation regions divided according to the target label library, wherein each annotation region includes a plurality of annotation labels having an order relationship;

[0208] A pre-annotation module 820 is used to pre-annotate the image to be annotated using a preset annotation model, so as to draw a plurality of annotation points in the image to be annotated and display an annotation label corresponding to each annotation point;

[0209] A second adjacent annotation label determination module 830 is used to traverse each annotation point and determine the adjacent annotation labels of the annotation label according to the annotation label of the currently traversed annotation point and the annotation area where the annotation point is located;

[0210] The second connection line drawing module 840 is used to draw a connection line between the annotation point corresponding to the adjacent annotation tag and the current annotation point.

[0211] A key point marking device provided in an embodiment of the present application can execute a key point marking method in Embodiment 2 of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0212] Embodiment 5

[0213] Fig. 9 A schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present application is shown in FIG. Fig. 9 As shown, the electronic device includes a processor 910, a memory 920, an input device 930, and an output device 940; the number of the processor 910 in the electronic device can be one or more. Fig. 9 A processor 910 is taken as an example; the processor 910, the memory 920, the input device 930 and the output device 940 in the electronic device can be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0214] The memory 920 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as program instructions / modules corresponding to the above-mentioned embodiments in the embodiments of the present application. The processor 910 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 920, that is, implements the methods mentioned in the above-mentioned method embodiments.

[0215] The memory 920 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 920 may further include a memory remotely arranged relative to the processor 910, and these remote memories may be connected to the device / terminal / server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0216] The input device 930 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 940 may include a display device such as a display screen.

[0217] Embodiment 6

[0218] Embodiment 6 of the present application also provides a storage medium containing computer executable instructions, and the computer executable instructions are used to execute the method in the above method embodiment when executed by a computer processor.

[0219] Of course, the storage medium containing computer executable instructions provided in the embodiments of the present application is not limited to the operations of the method described above, but can also execute related operations in the method provided in any embodiment of the present application.

[0220] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present application can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0221] It is worth noting that in the embodiment of the above-mentioned device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0222] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A key point labeling method, characterized in that: The method comprises: Acquire configuration data set for the image to be annotated, the configuration data comprising a target label library including a plurality of annotation labels and one or more annotation regions divided according to the target label library, wherein each annotation region comprises a plurality of annotation labels having an order relationship; Detecting a labeling operation performed by a user in the image to be labeled; Determine the target annotation label and the annotation position corresponding to the annotation operation, draw the corresponding target annotation point at the annotation position and display the corresponding target annotation label; In the target marking area where the target marking label is located, determining adjacent marking labels of the target marking label; If the adjacent annotation label has a corresponding annotation point in the image to be annotated, a connecting line is drawn between the annotation point and the target annotation point; After drawing a connecting line between the annotation point and the target annotation point, the method further includes: Creating a connection line instance of the connection line; The connecting line instance, the marked position of the target marked point and the target marked label are stored in association with each other and stored in the data field of the target marked point; Obtain the marking position of the marking point corresponding to the adjacent marking label; The adjacent annotation labels, the annotation positions of the annotation points corresponding to the adjacent annotation labels, and the connecting line instances are stored in association.

2. The method according to claim 1, characterized in that The connecting line instance includes a starting point position and an end point position, and the connecting line instance for creating the connecting line includes: If the adjacent annotation tag is the previous tag of the target annotation tag, the annotation position of the annotation point corresponding to the adjacent annotation tag is used as the starting position, and the annotation position of the target annotation point is used as the end position to establish a connecting line instance of the connecting line; and / or, If the adjacent annotation label is the next label of the target annotation label, the annotation position of the target annotation point is used as the starting position, and the annotation position of the annotation point corresponding to the adjacent annotation label is used as the end position to establish a connecting line instance of the connecting line.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: When it is detected that the target marking point is moved, determining the marking position of the target marking point after the movement; According to the moved marked position, the connecting line instance is updated; Redraw the corresponding connection line based on the updated connection line instance.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: When it is detected that the target annotation point is deleted, the connection line instance is deleted, and the corresponding connection line is deleted.

5. The method according to claim 1, characterized in that The method further comprises: Capturing various operations performed by the user in the image to be annotated, and generating operation objects according to the operations; The operation object is stored in a preset undo stack.

6. The method according to claim 1, 2 or 5, characterized in that: The configuration data further includes: a designated annotation tag for indicating a designated position, and drawing a corresponding target annotation point at the annotation position includes: If the target annotation label is not the designated annotation label, drawing a corresponding target annotation point at the annotation position according to a first preset label attribute; If the target annotation tag is the designated annotation tag, a corresponding target annotation point is drawn at the annotation position according to a second preset tag attribute.

7. The method according to claim 1, 2 or 5, characterized in that: The configuration data also includes connection line attributes set for each marked area respectively; The drawing of a connection line between the annotation point and the target annotation point includes: A connecting line between the marking point and the target marking point is drawn according to the connecting line attribute corresponding to the marking area where the target marking label is located.

8. The method according to claim 1 or 2, characterized in that: The configuration data also includes area visibility settings for each annotated area, and label visibility settings for the annotated labels; the method further includes: Visualizing the area visibility settings of each annotation area and the label visibility settings in the annotation page; When it is detected that the user triggers the area visibility setting for a certain annotation area, all annotation labels and corresponding annotation points of the annotation area are hidden in the image to be annotated; When it is detected that a user triggers the region visibility setting of a certain annotation region for one or more annotation tags in the annotation region, hiding the one or more annotation tags and the corresponding annotation points in the image to be annotated; When it is detected that the user triggers the tag visibility setting for a certain annotation area or one or more annotation tags, the annotation tags of the annotation area or the annotation points corresponding to the one or more annotation tags are hidden in the image to be annotated.

9. The method according to claim 1 or 2, characterized in that: The configuration data also includes a marking point type setting, and the method further includes: Visualizing the annotation point type setting in the annotation page; Detecting the type of the mark point set by the user for the target mark point, and recording the type of the mark point; The target annotation point is marked according to the annotation point type.

10. The method according to claim 1, characterized in that Before detecting the annotation operation performed by the user in the image to be annotated and the target annotation label set for the annotation operation, the method further includes: Opening a marking page, the marking page comprising a visible area for displaying the image to be marked; Acquire the size of the image to be annotated and the size of the visible area of ​​the annotated page; If the size of the image to be annotated is larger than the size of the visible area of ​​the annotation page, the size of the image to be annotated is adjusted according to the size of the visible area, and the adjusted image to be annotated is displayed in the visible area.

11. The method according to claim 10, characterized in that The annotation page also includes an annotation area display area for visually displaying each annotation area according to the configuration data, and the determining of the target annotation label and the annotation position corresponding to the annotation operation includes: Using the annotation tag selected by the user from the annotation area display area as the target annotation tag corresponding to the annotation operation; or, If the marking operation is the first marking operation in the current target marking area, the first marking label in the target marking area is used as the target marking label; If the marking operation is not the first marking operation in the current target marking area, the next unoccupied marking tag in the target marking area is selected as the target marking tag according to the marking tag corresponding to the previous marking operation; if there is no next unoccupied marking tag but there is an unoccupied marking tag before the marking tag corresponding to the previous marking operation, the target marking tag is determined by taking the modulus.

12. The method according to claim 1, characterized in that Also includes: When a submission instruction triggered by a user is detected, obtaining a labeling result of the user on the image to be labeled; The marking results are uploaded to the server.

13. A key point marking method, characterized in that: The method comprises: Acquire configuration data set for the image to be annotated, the configuration data comprising a target label library including a plurality of annotation labels and one or more annotation regions divided according to the target label library, wherein each annotation region comprises a plurality of annotation labels having an order relationship; Pre-annotating the image to be annotated using a preset annotation model, so as to draw a plurality of annotation points in the image to be annotated and display annotation labels corresponding to the annotation points; Traversing each annotation point, and determining the adjacent annotation labels of the annotation label according to the annotation label of the currently traversed annotation point and the annotation area where the annotation point is located; Draw a connecting line between the annotation point corresponding to the adjacent annotation label and the current annotation point; After drawing a connecting line between the annotation point corresponding to the adjacent annotation label and the current annotation point, the method further includes: Creating a connection line instance of the connection line; The connecting line instance, the marked position of the current marked point and the marked label corresponding to the current marked point are stored in association with each other and stored in the data field of the current marked point; Obtain the marking position of the marking point corresponding to the adjacent marking label; The adjacent annotation labels, the annotation positions of the annotation points corresponding to the adjacent annotation labels, and the connecting line instances are stored in association.

14. A key point marking device, characterized in that: The device comprises: A first configuration data acquisition module is used to acquire configuration data set for the image to be annotated, wherein the configuration data includes a target label library including a plurality of annotation labels and one or more annotation areas divided according to the target label library, wherein each annotation area includes a plurality of annotation labels having an order relationship; A labeling operation detection module, used to detect the labeling operation performed by the user in the image to be labeled; A target annotation point drawing module is used to determine the target annotation label and the annotation position corresponding to the annotation operation, and draw the corresponding target annotation point at the annotation position and display the corresponding target annotation label; A first adjacent annotation label determination module, configured to determine adjacent annotation labels of the target annotation label in a target annotation region where the target annotation label is located; A first connection line drawing module, configured to draw a connection line between a corresponding annotation point and a target annotation point if the adjacent annotation label has a corresponding annotation point in the image to be annotated; Wherein, the device further comprises: A connection line instance creation module, used to create a connection line instance of the connection line; An associated storage module is used to associate and store the connecting line instance, the marking position of the target marking point and the target marking label, and store them in the data field of the target marking point; and to obtain the marking position of the marking point corresponding to the adjacent marking label, and associate and store the adjacent marking label, the marking position of the marking point corresponding to the adjacent marking label and the connecting line instance.

15. A key point marking device, characterized in that: The device comprises: A second configuration data acquisition module is used to acquire configuration data set for the image to be annotated, wherein the configuration data includes a target label library including a plurality of annotation labels and one or more annotation areas divided according to the target label library, wherein each annotation area includes a plurality of annotation labels having an order relationship; A pre-annotation module, used to pre-annotate the image to be annotated using a preset annotation model, so as to draw a plurality of annotation points in the image to be annotated and display an annotation label corresponding to each annotation point; A second adjacent annotation label determination module is used to traverse each annotation point and determine the adjacent annotation labels of the annotation label according to the annotation label of the currently traversed annotation point and the annotation area where the annotation point is located; A second connection line drawing module is used to draw a connection line between the annotation point corresponding to the adjacent annotation label and the current annotation point; Wherein, the device further comprises: A connection line instance creation module, used to create a connection line instance of the connection line; An associated storage module is used to associate and store the connecting line instance, the marking position of the current marking point and the marking label corresponding to the current marking point, and store them in the data field of the current marking point; and to obtain the marking position of the marking point corresponding to the adjacent marking label, and associate and store the adjacent marking label, the marking position of the marking point corresponding to the adjacent marking label and the connecting line instance.

16. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 13.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images