A management method, device and system for marking tools

Abstract

The embodiment of the present application discloses a management method, device and system of a labeling tool, which is used in the field of artificial intelligence. The method includes: establishing a labeling project according to a labeling task; binding multiple labeling controls and / or multiple label sets to the labeling project; establishing an operation interface corresponding to the labeling project; the operation interface includes the multiple labeling controls and / or multiple label sets, and the operation interface is used to label the operation object according to the multiple labeling controls and / or label sets; in this way, the labeler can use multiple labeling controls to perform multi-task labeling on the same operation object under the same labeling project, thereby improving the labeling efficiency.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular, to a management method, device, and system for annotation tools. Background Art

[0002] A neural network is a complex network formed by a large number of interconnected neurons. It reflects many basic characteristics of the human brain and is a highly complex nonlinear dynamic learning system. As an operational model, each node of a neural network represents a specific output function, called an activation function, and the connection between each two nodes represents the weighted value of the connection signal to the next node, called the weight value. Therefore, the output of the neural network varies depending on the network's connection method, weight value, and activation function.

[0003] The training process of a neural network refers to the process of continuously optimizing the parameters of the neural network, that is, the process of making the neural network parameters continuously approach the correct parameters by iteratively calculating a large number of training samples; specifically, each training sample includes an input value and annotation information (correct output value), the input value of the training sample is input into the neural network, the model output value corresponding to the input value is obtained through the neural network calculation, and then the neural network parameters are updated through the gradient descent method based on the gap between the model output value and the annotation information. After a large number of iterative processes, the correct parameter values ​​are finally approached; therefore, correctly labeling the sample data and obtaining its annotation information has become the most basic link in neural network training.

[0004] The annotation system provides a large number of high-quality training samples for the training process of neural networks, which includes a variety of annotation controls. Different annotation controls are used for different annotation tasks. In the existing annotation system, since each annotation control is independent of each other, when annotating different tasks for the same sample, the annotator needs to constantly switch the pages corresponding to the annotation controls, and needs to create annotation projects under the operation pages of multiple annotation controls and complete the annotation in sequence. This will cause the annotator to perform a large number of switching and creation operations, resulting in low annotation efficiency. Therefore, how to optimize the annotation system and provide a more efficient and convenient annotation control operation interface has become an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a management method, device and system for a labeling tool, which is used to combine multiple labeling controls, so that labelers can use multiple labeling controls to label training samples under the same labeling project, thereby improving the labeling efficiency of training samples.

[0006] A first aspect of an embodiment of the present application provides a method for managing a marking tool, including:

[0007] When annotators perform multi-task annotation on training samples, they can create an annotation project in the annotation tool management system for the same training sample, aiming to perform multiple annotation operations on the training sample; then, bind multiple annotation controls and multiple label sets under the annotation project, and finally establish a corresponding interactive interface for the annotation project; optionally, multiple annotation controls and label sets can be displayed on the interactive interface, so that the annotator can perform multiple annotation operations on the training sample on the same interface.

[0008] The above-mentioned annotation system provides annotators with an interactive interface containing multiple annotation controls. When users perform multi-task annotation on training samples, they can perform annotation operations on the same interface without switching back and forth between the interfaces corresponding to multiple annotation controls. This improves the annotation efficiency of the annotators and increases the user experience.

[0009] In an optional implementation, according to the labeling requirements of multi-task labeling, multiple labeling activities can be created under the labeling project, and then, it is determined whether the labeling activity needs to be bound to a labeling control. If necessary, the labeling control is bound to the labeling activity, and a label set is bound to the labeling control; if the labeling control is not required, the label set is directly bound to the labeling activity for an overall description of the training sample, wherein the labeling control is generally used to determine the marked part of the training sample, and the label set is used to describe it, and the two together constitute the labeling information of the training sample.

[0010] Binding multiple annotation activities of the same training sample under the same annotation project can enable annotators to concentrate on multiple annotation activities on the training samples, thus improving the annotation efficiency of the annotators.

[0011] In an optional implementation, when the annotation system renders the operation interface corresponding to the annotation project according to the binding information, the annotator can input the relevant annotation information through the operation interface; specifically, the annotator can select the annotation control through the operation interface, and the system determines the target annotation control according to the user's selection operation, and then displays the operation interface corresponding to the target annotation control on the operation interface. In this way, the user can operate the training sample under the target annotation control and input instructions. The operation interface senses the user's input and determines the annotation information corresponding to the training sample according to the user's relevant operations.

[0012] In an optional implementation, when the annotator operates on a training sample under a certain annotation control, the operation interface needs to pop up the label set under the annotation control. The annotator then selects the label in the label set through the operation interface to generate the label of the training sample, and finally forms the annotation information of the training sample together with the annotation area under the annotation control.

[0013] In an optional implementation, the labels in the label set may include preset labels and custom labels, which may be in the form of an input box; the annotator may not only select a preset label as the label of the training sample, but also input a description of the training sample through the input box, and finally generate the labeling information of the training sample according to the content input by the annotator.

[0014] In an optional implementation, a network model may be bound to each annotation control, and the network model may generate a pre-label for the training sample according to the annotation operation of the annotator, and then finally generate the annotation information of the training sample through the pre-label.

[0015] A second aspect of an embodiment of the present application provides a management device for a marking tool, including:

[0016] A creation unit, used to create a labeling project according to a labeling task;

[0017] A processing unit, configured to bind a plurality of annotation controls and / or a plurality of tag sets to the annotation item;

[0018] The establishment unit is further used to establish an operation interface corresponding to the annotation item; the operation interface includes the multiple annotation controls and / or multiple label sets, and the operation interface is used to annotate the operation object according to the multiple annotation controls and / or label sets.

[0019] In an optional embodiment, the annotation project includes multiple annotation activities; the processing unit is specifically used to bind multiple annotation controls to the multiple annotation activities; a label set is bound to each of the annotation controls; wherein the annotation control is used to determine the marking position of the operation object, and the label set is used to describe the marking position.

[0020] In an optional embodiment, the management device of the annotation tool further includes a receiving unit and a determining unit, wherein the receiving unit is configured to receive a first selection operation of the user on the annotation control on the operation interface;

[0021] The determining unit is used to determine a target annotation control according to the first selection operation;

[0022] The receiving unit is further configured to receive a marking operation input for the operation object under the target marking control, wherein the marking operation corresponds to the target marking control;

[0023] The processing unit is further used to mark the operation object according to the marking operation.

[0024] In an optional implementation, the processing unit is specifically configured to display a tag set corresponding to the target annotation control according to the annotation operation;

[0025] The receiving unit is further configured to receive a second selection operation on the tag set corresponding to the target annotation control;

[0026] The determining unit is further configured to determine a target tag according to the second selection operation, and determine labeling information of the operation object according to the target tag.

[0027] In an optional implementation, the tag set includes a custom tag, and the receiving unit is further configured to receive an input operation on the custom tag;

[0028] The determining unit is further configured to determine the content of the custom tag according to the input operation; and determine the annotation information of the operation object according to the content of the custom tag.

[0029] In an optional embodiment, the processing unit is also used to bind a network model to the annotation control; the network model is used to determine the annotation information of the operation object; the operation object and the annotation operation are input into the network model; the target tag corresponding to the operation object is obtained through the network model; and the annotation information of the operation object is determined according to the target tag.

[0030] The third aspect of the present application provides a management system for a labeling tool, including an operation interface and a server; the server is used to: establish a labeling project according to a labeling task; bind multiple labeling controls and / or multiple label sets to the labeling project; establish an operation interface corresponding to the labeling project; the operation interface includes the multiple labeling controls and / or multiple label sets, and the operation interface is used to label the operation object according to the multiple labeling controls and / or label sets.

[0031] In an optional embodiment, the annotation project includes multiple annotation activities; the server is specifically used to: bind multiple annotation controls to the multiple annotation activities; bind a label set to each of the annotation controls; wherein the annotation control is used to determine the marking position of the operation object, and the label set is used to describe the marking position.

[0032] In an optional implementation, the operation interface is used to receive a first selection operation of the user on the annotation control;

[0033] The server is further configured to determine a target annotation control according to the first selection operation;

[0034] The operation interface is further used to receive a marking operation input for the operation object under the target marking control, and the marking operation corresponds to the target marking control;

[0035] The server is further used to mark the operation object according to the marking operation.

[0036] In an optional implementation, the operation interface is specifically used to display the tag set corresponding to the target tag control according to the tag operation; receive a second selection operation for the tag set corresponding to the target tag control;

[0037] The server is specifically configured to determine a target tag according to the second selection operation, and determine labeling information of the operation object according to the target tag.

[0038] In an optional implementation, the tag set includes a custom tag, and the operation interface is further used to receive an input operation on the custom tag;

[0039] The server is further configured to determine the content of the custom tag according to the input operation; and determine the annotation information of the operation object according to the content of the custom tag.

[0040] In an optional embodiment, the server is also used to: bind a network model to the annotation control; the network model is used to determine the annotation information of the operation object; input the operation object and the annotation operation into the network model; obtain the target tag corresponding to the operation object through the network model; and determine the annotation information of the operation object based on the target tag.

[0041] In a fourth aspect, the present application provides a chip or a chip system, the chip or the chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being used to run a computer program or instruction to perform a management method of a marking tool as described in any one of the first aspect to any one of the possible implementations of the first aspect;

[0042] The communication interface in the chip may be an input / output interface, a pin or a circuit.

[0043] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0044] The fifth aspect of the embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, it enables the computer to execute the management method of a marking tool described in the first aspect above.

[0045] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0046] In an embodiment of the present application, the annotation tool management system first establishes an annotation project according to the annotation task; then binds multiple annotation controls and / or multiple label sets to the annotation project, and finally renders the operation interface corresponding to the annotation project. The annotator can complete multiple annotation activities in the interface of the same annotation project without repeatedly creating projects or switching back and forth between the interfaces of multiple annotation controls. Instead, the operation objects are annotated under one annotation project, so that each operation object generates multiple labels, thereby improving the annotation efficiency of training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A system architecture diagram of a management system for a labeling tool provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of a method for managing a marking tool provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of an operation interface provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a flow chart of a method for displaying an operation interface provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a management device for a marking tool provided in this application;

[0052] Figure 6 A system architecture diagram of a management system for a marking tool provided in this application;

[0053] Figure 7 A schematic diagram of the structure of another management device of a marking tool provided in this application. DETAILED DESCRIPTION

[0054] The embodiments of the present application provide a management method, device and system for a labeling tool, which is used to combine multiple labeling controls, so that labelers can use multiple labeling controls to label training samples under the same labeling project, thereby improving the labeling efficiency of training samples.

[0055] The technical solution in the present application will be described in detail below in conjunction with the drawings in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0056] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] Artificial neural network (ANN), also known as neural network (NN), is a mathematical model or computational model that imitates the structure and function of biological neural networks. Neural networks are calculated by connecting a large number of artificial neurons. Generally, artificial neural networks can change their internal structure based on external information and are adaptive systems. Modern neural networks are a nonlinear statistical data modeling tool used to model complex relationships between inputs and outputs or to explore patterns in data.

[0058] As a computing model, neural network is composed of a large number of nodes (neurons) and the connections between them. Each node represents a specific output function, called an excitation function or activation function. The connection between each two nodes represents a weighted value, called a weight, which is equivalent to the memory of the artificial neural network. It can be understood that the output of the neural network varies depending on the network connection method, weight value and excitation function; and the network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.

[0059] The training process of a neural network refers to the process of continuously optimizing the parameters of the neural network, including the parameters of the activation function, weight values, etc.; by iteratively calculating a large number of training samples, the parameters of the neural network are constantly approaching the correct parameters. Specifically, each training sample includes annotation information (correct output value), and the training sample is input into the neural network. The model output value corresponding to the training sample is obtained through neural network calculation, and then the neural network parameters are updated through the gradient descent method according to the difference between the model output value and the annotation information; after a large number of iterative processes, each parameter is finally approached to the correct parameter value, and the calculation accuracy of the neural network is continuously improved. Therefore, correctly labeling the sample data and obtaining its annotation information has become the most basic link in neural network training.

[0060] Existing annotation systems provide a variety of annotation tools that can meet various annotation behaviors. Annotators can use a variety of annotation tools to add various labels (annotation information) to training samples. Exemplarily, annotation tools may include annotation controls and label sets; wherein the annotation controls may be drawing or marking tools used in the annotation activity; and the label sets may be information descriptions of training samples generated during the annotation activity.

[0061] Exemplarily, a neural network is a portrait classifier, and its purpose is to distinguish whether a portrait is smiling, then its corresponding training samples are multiple groups of portrait pictures; before training, the multiple groups of portrait pictures as training samples need to be marked, wherein the key points of the multiple groups of portrait pictures can be marked using graphic annotation controls, for example, the key points may include the corners of the eyes, the corners of the lips, the cheekbones, etc.; then the multiple groups of portrait pictures are described using a label set, including "smiling" and "not smiling"; finally, the training samples are input into the neural network, the output value of the neural network corresponding to the training samples and the error of its label are transmitted in reverse, and the parameters of the neural network are continuously adjusted so that the output value of the neural network gets closer and closer to the correct classification (label), thereby optimizing the classification performance of the neural network.

[0062] Annotation controls can be divided into image controls, text controls, audio controls, and video controls according to the different operation objects; image controls can include key point controls, rectangular box controls, and polyline controls, among which key point controls are used to mark image key points, rectangular box controls are used to mark local areas of images, and polyline controls are used to stroke key areas of images; text controls can include entity marking controls or relationship marking controls, among which entity marking controls are used to select part of a text in a paragraph, and relationship marking controls are used to record the relationship between text entities; audio controls can include breakpoint controls or interception controls, breakpoint controls are used to split a piece of audio, and interception controls are used to intercept part of the audio in a piece of audio; video controls include key point controls, rectangular box controls, polyline controls, etc., and their specific functions are similar to those of image controls, so they will not be described in detail. Annotation controls can meet various annotation needs, and annotators use annotation controls to mark each training sample.

[0063] In the existing annotation system, each annotation control is independent of each other. When using a certain annotation control to annotate a training sample, it is necessary to create an annotation project in the interface corresponding to the annotation control, and then use the annotation control to complete the annotation. However, for the same training sample, there are often multiple annotation tasks that appear in combination. Experiments have shown that for the same training sample, the more annotation information it contains, the more conducive it is to the training of the neural network. For example, in the above example, to train the neural network to judge the ability of a portrait to smile, not only can the key points such as the corners of the eyes, corners of the lips, and cheekbones of multiple groups of portrait photos be marked, but also the postures of the people in the multiple groups of portrait photos can be marked to assist the neural network in judging whether the people in the image are smiling. In this way, when annotating multiple groups of portrait photos, it is necessary not only to mark the key points, but also to select the body parts with rectangular frames, that is, key point controls and rectangular frame controls are needed to complete the annotation activities of the portrait pictures. Currently, annotators need to open the key point control page to create an annotation project, mark the key points of the portrait image and save them, then open the rectangular frame control page to create an annotation project, and import the portrait image again to select the area using the rectangular frame. This will cause a lot of redundant work for annotators, as they need to create multiple projects for the same training sample and switch between different projects, which seriously affects the annotation efficiency. Therefore, determining a more efficient annotation system to facilitate annotators to perform multiple annotation activities on the same training sample has become an urgent problem to be solved.

[0064] Based on the above problems, the present application provides a management system for annotation tools, which is used to combine multiple annotation controls to improve the annotation efficiency of annotators in performing multi-task annotation on the same training sample. Figure 1This is a system architecture diagram of a management system for a labeling tool provided in an embodiment of the present application, wherein the system architecture diagram includes a tool configuration module, a labeling interface display module, and a labeling interface storage module.

[0065] Among them, the tool configuration module is the key module in this application, which is used to combine multiple annotation controls. The user configures multiple different annotation tools in the tool configuration module, and then renders the annotation interface according to the tool configuration information, so that the annotator can use multiple annotation tools on one annotation interface.

[0066] The annotation interface display module renders the display and usage of multiple annotation controls based on the configuration information in the tool configuration module. It can also render the display and usage of multiple types of label sets based on the relationship between label sets and annotation controls, and provide them to the annotator's human-computer interaction interface.

[0067] The annotation interface saving module receives the annotated content of the annotator through the annotation interface, and saves the tags (annotated controls) and descriptions (label information) corresponding to the training samples; illustratively, it can record the tags of the controls on the samples in the form of key-value pairs and hang them under the controls; the label set also records the tags in the form of key-value pairs, and the controls and label sets are hung under the associated annotation activities, and multiple annotation activities are hung under one annotation project, so that structured data is used to save the annotation results.

[0068] Figure 2 A flowchart of a method for managing a marking tool provided in an embodiment of the present application; Figure 2 As shown, the management method includes the following steps:

[0069] 201. Establish a labeling project based on the labeling task.

[0070] When the annotator needs to annotate the training samples, he first creates an annotation project. For example, if the neural network trained by the training samples is a portrait classifier, which aims to analyze whether the faces in the portrait pictures are smiling, then the annotation task is to mark the training samples as "smile". Therefore, it is necessary to create an annotation project first. It can be understood that the annotation actions under this annotation project are all performed around whether the faces in the training samples are smiling.

[0071] The annotation system can provide a front-end interface, in which the annotator can input relevant instructions, so that the server corresponding to the annotation system can establish the annotation project according to the instructions and hang the training samples under the annotation project.

[0072] 202. Create multiple annotation activities under the annotation project.

[0073] After establishing the annotation project, you can establish multiple annotation activities under the annotation project. For example, in the above example, in order to make the neural network more accurately determine whether the face in the portrait picture is smiling, not only can the key points such as the corners of the eyes and the corners of the lips be marked, but also the age corresponding to the face, the body posture, etc. can be identified; it can be understood that this information can assist in determining whether a person is smiling. For example, by determining the age corresponding to the face, the curvature of the mouth corners of people of different ages when they smile can be determined. Through the body posture, it can be known that people in a jumping posture are more likely to be in a good mood and have a high possibility of smiling; therefore, under the annotation project, you may need to establish multiple annotation activities. For example, it may include three types of annotation activities: marking key points such as the corners of the eyes and the corners of the mouth; marking the face area; marking the portrait posture in the portrait picture.

[0074] 203. Determine in turn whether each annotation activity needs to be bound to a control; if so, execute step 204; if not, execute step 205.

[0075] When establishing multiple annotation activities, you need to bind corresponding tools to each annotation activity. First, you need to determine whether it needs to be bound to an annotation control based on the type of each annotation activity. If necessary, bind the corresponding control to it. If not, you can directly add a label set to annotate the training samples.

[0076] For example, when the annotation activity is to annotate the actions of a portrait in a portrait picture, an annotation control is needed to check the body parts of the portrait, so it is necessary to bind the annotation control; and when the annotation activity is to annotate the clarity of the portrait picture, it is not necessary to bind the annotation control, but directly bind the label set corresponding to the clarity. In this way, the annotator can select the corresponding number in the label set to directly describe the clarity of the portrait picture.

[0077] 204. Bind a labeling control to the labeling activity according to the type of the labeling activity.

[0078] When the annotation activity needs to be bound to an annotation control, the corresponding annotation control is selected for binding according to the type of the annotation activity; illustratively, in the example of step 202, three annotation controls can be bound to three annotation activities respectively, that is, the annotation activity for marking key points such as the corners of the eyes and the corners of the mouth is bound to the key point control; the annotation activity for marking the face area is bound to the rectangle control to lock the face area; the annotation activity for marking the portrait posture is bound to the polyline control to stroke the portrait body, etc.; it can be understood that one annotation project will be bound to multiple annotation controls.

[0079] 205. Bind tag set.

[0080] The annotation activity not only requires the participation of the annotation control, but also requires a label set for description, wherein each annotation control can correspond to a label set, and the label set includes multiple labels for determining the annotation information of the training sample. For example, in the example shown in step 204, the key point control can correspond to a label set, and the label set can include two labels, "smile" and "not smile"; the rectangle control can correspond to a label set about age, and the label set can include multiple labels such as "0-10 years old", "10 to 20 years old", and "20 to 30 years old"; the broken line control can correspond to a label set about posture, and can include multiple labels such as "jump", "stand", and "sitting". It can be understood that the above controls can also correspond to other types of label sets, and the setting of each annotation control corresponding to the label set is related to the annotation activity, and there is no specific limitation.

[0081] The label set can also be set individually. Generally, the label set set that is set individually is an overall description of the training sample, including the subject, style, overview, etc. of the training sample, and the specific form is not limited.

[0082] 206. Determine whether to bind the intelligent model. If so, execute step 207. If not, return to step 203.

[0083] Exemplarily, an intelligent model can also be bound to each annotation control. The label set bound to the annotation control in step 205 is generally manually preset, and the annotator is required to select labels in the label set to generate a label for a training sample. The intelligent model can automatically generate labels corresponding to the training samples based on the annotation controls, reducing the annotation work of the annotators and improving the efficiency of annotating the training samples.

[0084] 207. Bind an intelligent model to the annotation activity.

[0085] It is understandable that after the corresponding intelligent model is bound to the annotation activity, the training sample and the marking content of the annotation control can be input into the intelligent model, and the intelligent model obtains the label of the training sample through calculation. Exemplarily, one intelligent model can also be bound to multiple annotation activities, and multiple labels of the training sample can be obtained through the intelligent model, which is not specifically limited.

[0086] 208. Establish an operation interface corresponding to the annotation item.

[0087] After binding multiple annotation controls, label sets, and intelligent models to an annotation project, it is necessary to combine the multiple annotation controls and label sets and provide the corresponding interactive interface to the annotator.

[0088] In this interactive interface, annotators can operate multiple annotation controls, that is, they can complete multiple annotation activities in the interface of the same annotation project. There is no need to repeatedly create projects or switch back and forth between the interfaces of multiple annotation controls. Instead, they can annotate training samples in one annotation project, so that each training sample generates multiple labels, thereby improving the annotation efficiency of training samples.

[0089] Figure 3 A schematic diagram of the structure of an operation interface provided in an embodiment of the present application, such as Figure 3 As shown, the operation interface includes a control bar, an operation panel (displaying samples), a sample list, and a label set bar. When a labeling project is bound to multiple labeling controls, the server can create an operation interface for the labeling project, and the labeler can perform multi-task labeling on the training samples on the operation interface; Figure 4 A flowchart of a method for displaying an operation interface provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the display method includes:

[0090] 401. Display a control bar on the operation interface.

[0091] First, the operation interface of the annotation project can first display a control bar, which includes multiple annotation controls bound to the annotation project. Specifically, each annotation control can correspond to a button, and the annotator can select the annotation control by single-clicking or double-clicking the button.

[0092] 402. Receive a first selection operation on a label control in the control bar.

[0093] The annotation bar of the operation interface is used to receive the first selection operation input by the annotator, and can transmit the first selection operation to the server. The server determines the instruction input by the user through the received first selection operation, and determines the annotation control selected by the user based on the instruction.

[0094] 403. Determine a target annotation control according to the first selection operation.

[0095] 404. Receive a labeling operation for a training sample under the target labeling control.

[0096] After the target annotation control is selected, the operation panel provides the corresponding functions of the target annotation control. For example, when the rectangular control is selected, the operation panel can select a rectangular frame for the training sample. For example, the annotator selects a training sample from the sample list, imports the training sample into the operation panel, and then selects the rectangular control in the control bar. By inputting sliding operations and other actions on the operation panel, the rectangular frame is determined and the annotation area on the training sample is selected.

[0097] 405. Display the tag set corresponding to the target tag control in the tag set column according to the tag operation.

[0098] When the annotator marks the training sample using the selected target annotation control, the label set corresponding to the target annotation control will pop up in the label set bar. It can be understood that the display method of the labels in the label set may include radio button mode, that is, the annotator can only select one label from multiple labels, or multiple-select labels, that is, the annotator can select multiple labels in the label set; illustratively, the label set can also provide custom labels. After selecting the custom label, the annotator can enter the corresponding content, generate a label with the custom content, and use the custom content as the annotation information of the training sample. Specifically, the display method of the label set is not limited.

[0099] It can be understood that the annotation control in the control bar is associated with the display method of its corresponding tab set. For example, when the button of the annotation control is lit, its corresponding tab set button can also be lit. This makes it easier to display the association between the annotation control and the tab set and provide a better user experience.

[0100] 406. Receive a second selection operation on the tag set corresponding to the target annotation control.

[0101] Annotators can select labels from the label set to generate labels for training samples.

[0102] 407. Determine a target tag according to the second selection operation, and determine labeling information of the operation object according to the target tag.

[0103] The annotation information of the training samples can be generated according to the content of the selected labels.

[0104] Figure 5 A structural diagram of a management device for a marking tool provided in this application, such as Figure 5 As shown, the marking tool management device includes:

[0105] An establishing unit 501 is used to establish a labeling project according to the labeling task;

[0106] A processing unit 502 is used to bind a plurality of annotation controls and / or a plurality of tag sets to the annotation item;

[0107] The establishing unit 501 is further used to establish an operation interface corresponding to the annotation item; the operation interface includes the multiple annotation controls and / or multiple label sets, and the operation interface is used to annotate the operation object according to the multiple annotation controls and / or label sets.

[0108] In an optional embodiment, the annotation project includes multiple annotation activities; the processing unit 502 is specifically used to bind multiple annotation controls to the multiple annotation activities; a label set is bound to each of the annotation controls; wherein the annotation control is used to determine the marking position of the operation object, and the label set is used to describe the marking position.

[0109] In an optional implementation, the management device of the annotation tool further includes a receiving unit 503 and a determining unit 504, wherein the receiving unit 503 is configured to receive a first selection operation of the user on the annotation control on the operation interface;

[0110] The determining unit 504 is used to determine a target annotation control according to the first selection operation;

[0111] The receiving unit 503 is further configured to receive a marking operation input for the operation object under the target marking control, wherein the marking operation corresponds to the target marking control;

[0112] The processing unit 502 is further configured to mark the operation object according to the marking operation.

[0113] In an optional implementation, the processing unit 502 is specifically configured to display a tag set corresponding to the target annotation control according to the annotation operation;

[0114] The receiving unit 503 is further configured to receive a second selection operation on the tag set corresponding to the target annotation control;

[0115] The determining unit 504 is further configured to determine a target tag according to the second selection operation, and determine labeling information of the operation object according to the target tag.

[0116] In an optional implementation, the tag set includes a custom tag, and the receiving unit 503 is further configured to receive an input operation on the custom tag;

[0117] The determining unit 504 is further configured to determine the content of the custom tag according to the input operation; and determine the annotation information of the operation object according to the content of the custom tag.

[0118] In an optional embodiment, the processing unit 502 is also used to bind a network model to the annotation control; the network model is used to determine the annotation information of the operation object; the operation object and the annotation operation are input into the network model; the target tag corresponding to the operation object is obtained through the network model; and the annotation information of the operation object is determined according to the target tag.

[0119] Figure 6 A system architecture diagram of a management system for a marking tool provided in this application, such as Figure 6 As shown, the management system of the marking tool includes an operation interface 601 and a server 602 .

[0120] The server 602 is used to: establish a labeling project according to the labeling task; bind multiple labeling controls and / or multiple label sets to the labeling project; establish an operation interface corresponding to the labeling project; the operation interface includes the multiple labeling controls and / or multiple label sets, and the operation interface is used to label the operation object according to the multiple labeling controls and / or label sets.

[0121] In an optional embodiment, the annotation project includes multiple annotation activities; the server 602 is specifically used to: bind multiple annotation controls to the multiple annotation activities; bind a label set to each of the annotation controls; wherein the annotation control is used to determine the marking position of the operation object, and the label set is used to describe the marking position.

[0122] In an optional implementation, the operation interface 601 is used to receive a first selection operation of the user on the annotation control;

[0123] The server 602 is further configured to determine a target annotation control according to the first selection operation;

[0124] The operation interface 601 is further used to receive a marking operation input for the operation object under the target marking control, and the marking operation corresponds to the target marking control;

[0125] The server 602 is further configured to mark the operation object according to the marking operation.

[0126] In an optional implementation, the operation interface 601 is specifically used to display the label set corresponding to the target labeling control according to the labeling operation; receive a second selection operation for the label set corresponding to the target labeling control;

[0127] The server 602 is specifically configured to determine a target tag according to the second selection operation, and determine annotation information of the operation object according to the target tag.

[0128] In an optional implementation, the tag set includes a custom tag, and the operation interface 601 is further used to receive an input operation on the custom tag;

[0129] The server 602 is further configured to determine the content of the custom tag according to the input operation; and determine the annotation information of the operation object according to the content of the custom tag.

[0130] In an optional embodiment, the server 602 is also used to: bind a network model to the annotation control; the network model is used to determine the annotation information of the operation object; input the operation object and the annotation operation into the network model; obtain the target tag corresponding to the operation object through the network model; and determine the annotation information of the operation object according to the target tag.

[0131] See also Figure 7 , is a structural diagram of another annotation tool management device 700 provided in an embodiment of the present application. The annotation tool management device 700 includes: a processor 701, a memory 702, and a communication interface 703.

[0132] The processor 701, the memory 702, and the communication interface 703 are interconnected via a bus; the bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0133] The memory 702 may include a volatile memory, such as a random-access memory (RAM); the memory may also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 702 may also include a combination of the above types of memory.

[0134] The processor 701 may be a central processing unit (CPU), a network processor (NP) or a combination of a CPU and a NP. The processor 701 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0135] The communication interface 703 may be a wired communication interface, a wireless communication interface or a combination thereof, wherein the wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface or a combination thereof. The wireless communication interface may be a WLAN interface, a cellular network communication interface or a combination thereof, etc.

[0136] The processor 701 is used to run the computer program or instructions in the memory 702 to perform Figure 2 A method for managing a marking tool as described in any one of the possible implementations of the illustrated embodiment and Figure 4 A method for displaying an operation interface as described in any one of any possible implementations of the illustrated embodiment;

[0137] The embodiment of the present application also provides a chip or a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used to run a computer program or instruction to perform Figure 2 A method for managing a marking tool as described in any one of the possible implementations of the illustrated embodiment and Figure 4 A method for displaying an operation interface as described in any one of any possible implementations of the illustrated embodiment;

[0138] The communication interface in the chip may be an input / output interface, a pin or a circuit.

[0139] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0140] An embodiment of the present application also provides a computer storage medium for storing computer software instructions used for the above-mentioned management device based on a marking tool, which includes a program designed for executing a management device for a marking tool.

[0141] An embodiment of the present application further provides a computer program product, which includes computer software instructions. The computer software instructions can be loaded by a processor to implement the process in the above-mentioned management method of a marking tool.

[0142] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A method for managing a marking tool, characterized in that: The method comprises: Establishing a labeling project according to the labeling task, wherein the labeling project includes multiple labeling activities, and the multiple labeling activities belong to the same training sample; Binding a plurality of annotation controls and / or a plurality of label sets to the annotation project, comprising: binding an annotation control or a label set to each annotation activity according to the type of each annotation activity in the plurality of annotation activities, and binding a label set to each of the annotation controls; Establishing an operation interface corresponding to the annotation item; the operation interface includes the multiple annotation controls and / or multiple label sets, and the operation interface is used to annotate the operation object according to the multiple annotation controls and / or label sets; In response to annotating the operation object by using the target annotation control, the operation interface pops up a label set corresponding to the target annotation control, and the label set corresponding to the target annotation control is in a binding relationship with the target annotation control.

2. The method according to claim 1, characterized in that The annotation control is used to determine the marking position of the operation object, and the label set is used to describe the marking position.

3. The method according to claim 1, characterized in that The method further comprises: Receiving a first selection operation of the annotation control by the user on the operation interface; Determining the target annotation control according to the first selection operation; receiving a marking operation input for the operation object under the target marking control, wherein the marking operation corresponds to the target marking control; The operation object is labeled according to the labeling operation.

4. The method according to claim 3, characterized in that The marking the operation object according to the marking operation includes: Displaying a label set corresponding to the target annotation control according to the annotation operation; Receiving a second selection operation on the tag set corresponding to the target annotation control; A target label is determined according to the second selection operation, and labeling information of the operation object is determined according to the target label.

5. The method according to claim 4, characterized in that The tag set includes a custom tag, and the method further includes: Receiving an input operation on the custom tag; Determine the content of the custom tag according to the input operation; The labeling information of the operation object is determined according to the content of the custom tag.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: Binding a network model to the annotation control; the network model is used to determine the annotation information of the operation object; The marking the operation object according to the marking operation includes: Inputting the operation object and the annotation operation into the network model; Obtaining a target label corresponding to the operation object through the network model; Determine the labeling information of the operation object according to the target label.

7. A management system for annotation tools, characterized in that: The system includes an operation interface and a server; the server is used to: Establishing a labeling project according to the labeling task, wherein the labeling project includes multiple labeling activities, and the multiple labeling activities belong to the same training sample; Binding a plurality of annotation controls and / or a plurality of label sets to the annotation project, comprising: binding an annotation control or a label set to each annotation activity according to the type of each annotation activity in the plurality of annotation activities, and binding a label set to each of the annotation controls; Establishing an operation interface corresponding to the annotation item; the operation interface includes the multiple annotation controls and / or multiple label sets, and the operation interface is used to annotate the operation object according to the multiple annotation controls and / or label sets; In response to annotating the operation object by using the target annotation control, the operation interface pops up a label set corresponding to the target annotation control, and the label set corresponding to the target annotation control is in a binding relationship with the target annotation control.

8. The system according to claim 7, characterized in that The annotation control is used to determine the marking position of the operation object, and the label set is used to describe the marking position.

9. The system according to claim 7, characterized in that The operation interface is used to receive a first selection operation of the user on the annotation control; The server is further configured to determine the target annotation control according to the first selection operation; The operation interface is further used to receive a marking operation input for the operation object under the target marking control, and the marking operation corresponds to the target marking control; The server is further used to mark the operation object according to the marking operation.

10. The system according to claim 9, characterized in that The operation interface is specifically used for: Displaying a label set corresponding to the target annotation control according to the annotation operation; Receiving a second selection operation on the tag set corresponding to the target annotation control; The server is specifically configured to determine a target tag according to the second selection operation, and determine labeling information of the operation object according to the target tag.

11. The system according to claim 10, characterized in that The label set includes custom labels, and the operation interface is also used for: Receiving an input operation on the custom tag; The server is further configured to determine the content of the custom tag according to the input operation; and determine the annotation information of the operation object according to the content of the custom tag.

12. The system according to any one of claims 9 to 11, characterized in that The server is also used for: Binding a network model to the annotation control; the network model is used to determine the annotation information of the operation object; Inputting the operation object and the annotation operation into the network model; Obtaining a target label corresponding to the operation object through the network model; Determine the labeling information of the operation object according to the target label.

13. A computer-readable storage medium storing one or more computer-executable instructions, characterized in that: When the computer-executable instructions are executed by a processor, the processor performs the method according to any one of claims 1 to 6.

Images