Data annotation management method and device, electronic equipment and storage medium

Abstract

The invention provides a data annotation management method and device, electronic equipment and a storage medium, the method is applied to an artificial intelligence platform, and the method comprises the following steps: in response to a data annotation task request of a creation user, determining first access account information of an annotation component; based on the first access account information, obtaining a session identifier of the annotation component; logging in the annotation component based on the session identifier of the annotation component; creating a data labeling task by utilizing a labeling component based on the data labeling task request; synchronizing the to-be-labeled data in the artificial intelligence platform to the data labeling task based on the access interface of the labeling component; and automatically annotating the to-be-annotated data in the data annotation task. According to the method, the user does not need to input identity information to log in the labeling component, password-free login of the labeling component by the user is realized, the convenience of user operation is improved, automatic labeling of the to-be-labeled data is realized, and the data labeling efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a data annotation management method, device, electronic device and storage medium. Background Art

[0002] With the vigorous development of artificial intelligence-related industries, more and more scientific research companies and university researchers need massive amounts of labeled data when training and building models. The current mainstream artificial intelligence platforms usually do not support data labeling functions. When users of artificial intelligence platforms have data labeling tasks, they are required to enter their identity information and log in to a dedicated labeling platform, which makes user operations cumbersome. In addition, for artificial intelligence platforms that have data labeling functions, they usually also require manual data labeling, and the data labeling efficiency is low. Summary of the invention

[0003] The present disclosure provides a data annotation management method, device, electronic device and storage medium, which do not require users to input identity information to log in to the annotation component, thus realizing password-free login of the annotation component, improving the convenience of user operation, and realizing automatic annotation of the data to be annotated, thereby improving data annotation efficiency.

[0004] The first aspect of the present disclosure proposes a data annotation management method, which is applied to an artificial intelligence platform, including: in response to a request to create a data annotation task for a user, determining first access account information of an annotation component; based on the first access account information, obtaining a session identifier of the annotation component; based on the session identifier of the annotation component, logging into the annotation component; based on the data annotation task request, creating a data annotation task using the annotation component; based on an access interface of the annotation component, synchronizing the data to be annotated in the artificial intelligence platform to the data annotation task; and automatically annotating the data to be annotated in the data annotation task.

[0005] The second aspect embodiment of the present disclosure proposes an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when being used to run the computer program, executes the method described in the first aspect embodiment of the present disclosure.

[0006] The third aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer commands, wherein the computer commands are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0007] In summary, a data annotation management method provided by the present disclosure is applied to an artificial intelligence platform, including: in response to a request to create a data annotation task for a user, determining the first access account information of the annotation component; based on the first access account information, obtaining the session identifier of the annotation component; based on the session identifier of the annotation component, logging into the annotation component; based on the data annotation task request, using the annotation component to create a data annotation task; based on the access interface of the annotation component, synchronizing the data to be annotated in the artificial intelligence platform to the data annotation task; and automatically annotating the data to be annotated in the data annotation task. The method of the present disclosure obtains the session identifier of the annotation component so that the artificial intelligence platform can log in to the above-mentioned annotation component using the session identifier without the user having to input identity information, thereby realizing the user's password-free login to the annotation component and improving the convenience of user operation; and realizing automatic annotation of the data to be annotated through the artificial intelligence platform, thereby improving the efficiency of data annotation.

[0008] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Figure 1 A flowchart of a data annotation management method provided by an embodiment of the present disclosure; Figure 2 A flowchart of a data annotation management method provided by an embodiment of the present disclosure; Figure 3 This is a flowchart of another data annotation management method provided by an embodiment of the present disclosure; Figure 4 A flowchart of another data annotation management method provided by an embodiment of the present disclosure; Figure 5 A flowchart of another data annotation management method provided by an embodiment of the present disclosure; Figure 6 A flowchart of another data annotation management method provided by an embodiment of the present disclosure; Figure 7 A flowchart of another data annotation management method provided by an embodiment of the present disclosure; Figure 8 A flowchart of another data annotation management method provided by an embodiment of the present disclosure; Fig. 9 This is a flowchart of another data annotation management method provided by an embodiment of the present disclosure; Fig.10 A flowchart of another data annotation management method provided by an embodiment of the present disclosure; Fig.11 A flowchart of another data annotation management method provided by an embodiment of the present disclosure; Fig.12 This is a flowchart of another data annotation management method provided by an embodiment of the present disclosure; Fig.13 A schematic diagram of the structure of a data annotation management device provided by an embodiment of the present disclosure; Fig.14 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0010] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0011] With the vigorous development of artificial intelligence-related industries, more and more scientific research companies and university researchers need massive amounts of labeled data when training and building models. The current mainstream artificial intelligence platforms usually do not support data labeling functions. When users of artificial intelligence platforms have data labeling tasks, they are required to enter their identity information and log in to a dedicated labeling platform, which makes user operations cumbersome. In addition, for artificial intelligence platforms that have data labeling functions, they usually also require manual data labeling, and the data labeling efficiency is low.

[0012] In order to solve the technical problems existing in the related art, an embodiment of the present disclosure provides a data annotation management method.

[0013] The embodiments of the present disclosure will be described in detail below.

[0014] like Figure 1 As shown, an embodiment of the present disclosure provides a data annotation management method, which is applied to an artificial intelligence platform and includes the following steps: Step 101, in response to a request to create a data annotation task for a user, determine first access account information of an annotation component.

[0015] In some embodiments, when the creating user initiates a data labeling task request on the artificial intelligence platform for the first time, the artificial intelligence platform will create an account information for the creating user to log in to the labeling component, i.e., the first access account information, and store the first access account information in the storage table or database corresponding to the artificial intelligence platform.

[0016] In some embodiments, when the creating user is not initiating a data labeling task request on the artificial intelligence platform for the first time (i.e., the creating user already has account information for logging into the labeling component), the artificial intelligence platform can query the corresponding storage table or database through the backend service to determine the first access account information corresponding to the creating user.

[0017] In some embodiments, data can be transmitted between the artificial intelligence platform and the annotation component through an access interface, such as through an Application Programming Interface (API). Therefore, in order for the artificial intelligence platform to successfully log in to the annotation component using the first access account information, the first access account information may include at least one of the following: the login name of the created user, the login password of the created user, and information about the access interface between the artificial intelligence platform and the annotation component.

[0018] In some embodiments, the labeling component may be a labelU-Kit labeling component, a LabelStudio component, or other component that can be used for data labeling.

[0019] In some embodiments, to facilitate the creation and storage of the first access account information, the login name of the created user may be a rewrite of the user name used by the created user to log in to the artificial intelligence platform (taking the labeling component as the LabelStudio component as an example, since the LabelStudio component requires that the login name must be in an email format, an email suffix such as @label.com may be added to the user name used to log in to the artificial intelligence platform to generate the login name of the created user), the login password of the created user may be a randomly generated 8-bit character, and the information of the access interface between the artificial intelligence platform and the labeling component may be a randomly generated 32-bit character, but is not limited to this. It should be understood that the specific form of the above-mentioned first access account information is only an example and may also be other forms.

[0020] Step 102: Obtain a session identifier of the annotation component based on the first access account information.

[0021] In some embodiments, the artificial intelligence platform can use the obtained first access account information to access the first resource address of the annotation component through the backend microservice of the artificial intelligence platform, so that when the artificial intelligence platform successfully accesses the first resource address, it receives the cross-site request information sent by the annotation component; and then uses the cross-site request information and the first access account information to access the annotation component, so that when the annotation component is successfully accessed, it receives the session identifier sent by the annotation component through the backend microservice of the artificial intelligence platform.

[0022] In some embodiments, the session identifier is used to prevent sensitive data in the first access account information (such as the login name of the creating user, the login password of the creating user, etc.) from being intercepted when the artificial intelligence platform uses the first access account information to log in to the annotation component.

[0023] In some embodiments, the first resource address may be a Uniform Resource Locator (URL) address of the annotation component.

[0024] In some embodiments, the session identifier may be a sessionid.

[0025] Step 103: Log in to the annotation component based on the session identifier of the annotation component.

[0026] In some embodiments, the artificial intelligence platform can log in to the annotation component from the front-end service of the artificial intelligence platform using the first domain name proxy module based on the session identifier of the annotation component. The first domain name proxy module corresponds to the front-end service of the artificial intelligence platform (that is, the first domain name proxy module can be integrated in the front-end microservice), and the annotation component includes a second domain name proxy module corresponding to the first domain name proxy module.

[0027] In other words, the artificial intelligence platform can pass the session identifier and the first access account information to the second domain name proxy module in the annotation component through the first domain name proxy module in the front-end microservice to realize the transmission of the identity information of the created user, and then use the second domain name proxy module to transmit the login name of the created user and the login password of the created user contained in the first access account information to the login system of the annotation component, thereby realizing the password-free login of the created user to the annotation component.

[0028] In some embodiments, the first domain name proxy module and the second domain name proxy module may be niginx modules, but are not limited thereto. Any module that can implement cross-domain transmission of sensitive information may be the first domain name proxy module or the second domain name proxy module.

[0029] In some embodiments, the artificial intelligence platform can execute a first command based on the session identifier of the annotation component to call the remote client component and log in to the annotation component from the backend service of the artificial intelligence platform. The remote client component corresponds to the backend service of the artificial intelligence platform, and the annotation component includes a remote server component corresponding to the remote client component.

[0030] In other words, the artificial intelligence platform can execute the first command in the remote client component, thereby calling the remote client component and passing the session identifier from the port of the artificial intelligence platform to the port of the annotation component. When the second domain name proxy module in the annotation component listens to the port of the annotation component and detects that the port has successfully received the session identifier, the session identifier can be sent to the remote server component of the annotation component, thereby using the remote server component to pass the session identifier to the login system of the annotation component, thereby bypassing the artificial intelligence platform and the front end of the annotation component and realizing password-free login to the annotation component from the back-end microservice.

[0031] In some embodiments, the remote client component can be a jsonrpc module integrated in the back-end service of the artificial intelligence platform, and the remote measurement server component can be a jsonrpc module integrated in the annotation component, but is not limited to this. It can also be other modules that can realize session identifier transmission, which is not limited in this disclosure.

[0032] In some embodiments, the first command may be a Python statement, but is not limited thereto, and may also be other programming statements capable of calling remote client components. It should be understood that when using the first statement, both the artificial intelligence platform and the annotation component are configured with a language environment corresponding to the first statement.

[0033] Step 104: Based on the data annotation task request, create a data annotation task using the annotation component.

[0034] In some embodiments, when the artificial intelligence platform successfully logs in to the annotation component using the first access account information, the artificial intelligence platform can create a data annotation task in the annotation component according to the relevant information in the data annotation task request.

[0035] In some embodiments, the creating user may also create a labeling task in the labeling component by selecting a labeling directory in the artificial intelligence platform.

[0036] In some embodiments, the data labeling task request may include information such as the task name, data to be labeled, and labeling type (such as text, picture, video, etc.), but is not limited to this. It may also be used only to create a data labeling task item in the labeling component without including information such as the data to be labeled.

[0037] Specifically, the processor resources used to create the data labeling task can be determined based on the candidate resource group corresponding to the creation user in the artificial intelligence platform, wherein the candidate resource group belongs to the artificial intelligence platform; and then based on the processor resources and the data path of the data to be labeled corresponding to the data labeling task request, the data labeling task is created in the labeling component.

[0038] In some embodiments, the candidate resource group may be CPU resources and / or GPU resources corresponding to the artificial intelligence platform.

[0039] In some embodiments, the data annotation task (ie, the annotation page) created in the annotation component may include information such as the task name, the data to be annotated, and the annotation type, but is not limited thereto.

[0040] In some embodiments, after successfully creating a data annotation task in the annotation component, the program programming API interface between the annotation component and the artificial intelligence platform can be used to display the data in the preset display area of ​​the artificial intelligence platform. Figure 2 The resource configuration, data configuration, data annotation list, annotation task details, and other information for creating a data annotation task are shown.

[0041] In some embodiments, the corresponding relationship between the artificial intelligence platform and the annotation component can be as follows: Figure 3 shown.

[0042] Specifically, the resource configuration may include the resource group of the artificial intelligence platform corresponding to the data labeling task, the network type, and the CPU or GPU used to create the data labeling task; the data configuration may include the directory of the labeled data (i.e., the above-mentioned data to be labeled) in the data labeling task.

[0043] The data annotation list can include the task name, status, running time, resource information, creation time, annotation page, and the running node (i.e., running node) used by the artificial intelligence platform; the creating user can perform tasks Name search, node search, user search and other functions in the data annotation list, and the creating user can also perform tasks deletion, task stop, task start and other operations in the data annotation list.

[0044] When there is a data labeling task, the labeling task details can include specific task labeling projects (project 1, project 2, etc.), the total number of data to be labeled, the number of labeled data to be labeled, and the data synchronization list. The creating user can perform data synchronization operations between the artificial intelligence platform and the labeling component in the labeling task details interface, and can also export the results of the labeled data (for example, exporting data labeling results in JSON format, but not limited to this); when there is no data labeling task or the data labeling task has been completed, the labeling task details can provide options such as creating task projects, creating labeling tools, local data synchronization, jumping to the labeling component, etc., which are convenient for creating users to get started quickly.

[0045] In some embodiments, when the annotation component successfully creates a data annotation task, the data structure stored in the data annotation task may be as follows:

[0046] Table 1 It should be understood that the above description is only an example and should not limit the present disclosure. The artificial intelligence platform can realize some or all of the above functions, and can also realize other functions besides the above description; the storage structure of the data labeling task may also differ from the content shown in Table 1, and the present disclosure is not limited to this.

[0047] Step 105: Based on the access interface of the annotation component, synchronize the data to be annotated in the artificial intelligence platform to the data annotation task.

[0048] In some embodiments, the access interface of the annotation component may be an API interface of the annotation component, but is not limited thereto.

[0049] In some embodiments, when the task labeling request does not include the data to be labeled, the artificial intelligence platform can synchronize the data to be labeled uploaded by the creating user to the artificial intelligence platform or the creating user's local data to the data labeling task through an access interface to realize data flow between the artificial intelligence platform and the labeling component.

[0050] Step 106: automatically label the data to be labeled in the data labeling task.

[0051] In some embodiments, the artificial intelligence platform can be communicated and connected with multiple computing nodes. The intelligent scheduling system of the artificial intelligence platform (such as the Kubernetes system, etc.) can allocate computing nodes for automatic user labeling to the data labeling task based on information such as the amount of data to be labeled, the type of data to be labeled, and the user level of the creating user contained in the data labeling task, and then label the data to be labeled by calling the inference model in the allocated computing node, thereby realizing automatic labeling of the data to be labeled.

[0052] In some embodiments, Figure 4 As shown, in addition to being connected to computing power nodes, the artificial intelligence platform can also be connected to training nodes and training and computing power fusion nodes. The training nodes are used to train the inference model so that the inference model can realize the function of automatic labeling. The training and computing power fusion nodes are used to train the inference model and use the trained inference model to automatically label the data to be labeled in the data labeling task.

[0053] In summary, the method disclosed herein obtains the session identifier of the annotation component so that the artificial intelligence platform can use the session identifier to log in to the above-mentioned annotation component without the user having to enter identity information, thereby realizing password-free login of the annotation component for the user and improving the convenience of user operation; and realizes automatic annotation of the data to be annotated through the artificial intelligence platform, thereby improving the efficiency of data annotation.

[0054] Figure 5 A flow chart of a data annotation management method proposed in the present disclosure is further shown. Figure 1 The illustrated embodiment further explains step 102. Figure 5 The following steps may be included.

[0055] Step 501: Access a first resource address of a marking component based on first access account information.

[0056] In some embodiments, the URL address (i.e., the first resource address) of the annotation component can be accessed through the backend service of the artificial intelligence platform using the login name and login password of the creating user in the first access account information.

[0057] Step 502: When the first resource address is successfully accessed, receive cross-site request information sent by the annotation component.

[0058] In some embodiments, when the artificial intelligence platform successfully accesses the first resource address, the annotation component returns cross-site request information to the artificial intelligence platform so that the artificial intelligence platform can send sensitive information to the annotation component across sites. At this time, the artificial intelligence platform receives the cross-site request information sent by the annotation component.

[0059] In some embodiments, the cross-site request information is, for example, csrfmiddlewaretoken, but is not limited thereto.

[0060] Step 503: Based on the first access account information and the cross-site request information, obtain a session identifier of the annotation component.

[0061] In some embodiments, the artificial intelligence platform can use the login name and login password of the user who created the first access account, as well as the cross-site request information to log in to the annotation component. When the artificial intelligence platform successfully logs in to the annotation component, the annotation component will store the access credentials of the artificial intelligence platform (such as the first access account information and cross-site request information, etc.), and return a session identifier to the artificial intelligence platform through the browser, so that the artificial intelligence platform can realize the transmission of sensitive information between browsers, solve the browser cross-domain problem between the artificial intelligence platform and the annotation, and lay the foundation for password-free login to the annotation component.

[0062] In some embodiments, the session identifier is, for example, a sessionid field, but is not limited thereto.

[0063] In summary, the data annotation management method proposed in the present disclosure is applied to an artificial intelligence platform, including: accessing the first resource address of the annotation component based on the first access account information; when the first resource address is successfully accessed, receiving the cross-site request information sent by the annotation component; based on the first access account information and the cross-site request information, obtaining the session identifier of the annotation component. The method of the present disclosure, by using the first access account information to determine the session identifier of the annotation component, lays the foundation for realizing the user's password-free login to the annotation component.

[0064] Figure 6 A flow chart of a data annotation management method proposed in the present disclosure is further shown. Figure 1 The embodiment shown further explains step 103. Figure 6 The following steps may be included: Step 601, using the first domain name proxy module, maps the domain name of the artificial intelligence platform to the domain name address of the second domain name proxy module of the annotation component.

[0065] In some embodiments, a first domain name proxy module can be used to map the domain name (https) of the artificial intelligence platform to the domain name (https) address of the second domain name proxy module, so that the artificial intelligence platform can send the first access account information and session identifier to the annotation component through the browser.

[0066] Step 602: Based on the domain name address, the first access account information and the session identifier are sent to the annotation component to log in to the annotation component.

[0067] In some embodiments, to ensure the security of the first access account information, the Advanced Encryption Standard (AES) can be used to set encryption key information for the first access account information, for example, the information is set to annotationslabelstudio==, and the first access account information is encrypted and decrypted through the encrypted information, but is not limited to this, and other encryption methods may also be used, which is not limited to the present disclosure.

[0068] In some embodiments, the first access account information and the session identifier may be sent to the annotation component through a browser according to the domain name address of the second domain name proxy module, thereby enabling password-free login of the annotation component for the creating user.

[0069] For example, in an optional embodiment, it can be as follows Figure 7 As shown, the artificial intelligence platform and other front-end password-free login annotation components include the following steps: 1. The AI ​​platform obtains the username and password for logging into the labeling component Labelstudio. 2. The AI ​​platform uses the obtained username and password to query the labeling component container through the backend service, and has parsed and obtained the cross-site request information of the login component. 3. Since the password may be encrypted, the AES algorithm can be used to decrypt the password, and the username, the password decrypted by AES, and the extended request information can be used to access the labeling component, and the user login access credentials are saved in the labeling component. 4. Through the session identifier returned by the labeling component, the AI ​​platform obtains the session identifier and the first resource address (i.e., the URL of the labeling component). 5. Using the front-end service of the AI ​​platform, use the session identifier to access the first resource address. 6. Use the first domain name proxy module (nginx of the AI ​​platform) in the front-end service to perform domain name mapping to map the domain name of the AI ​​platform to the second domain name proxy module (nginx of the labeling component). 7. The session identifier, username, and password are packaged into a small text (cookie) through the first domain name proxy module and sent to the second domain name proxy module, thereby realizing password-free login to the labeling component.

[0070] In other words, first, in the backend microservice of the artificial intelligence platform, obtain the user and password for logging in to access the annotation component, then access the URL address of the annotation component in the backend microservice, obtain the cross-site request information csrfmiddlewaretoke, use the login user and password, and the cross-site request information to log in to access the annotation component, and after successfully logging in to the annotation component, the microservice of the annotation component will store the user's access credentials, and at the same time return the session identifier sessionid field to the backend service of the artificial intelligence platform in the browser; if the session identifier is passed in the front-end microservice of the artificial intelligence platform, it is necessary to solve the browser cross-domain problem, so the first domain name proxy module ngnix is ​​installed in the component of the front-end microservice to perform domain name proxy and information transmission; at the same time, the second domain name proxy module ngnix also needs to be installed in the annotation component microservice, and the corresponding keys (for example, certifaicate parameters and key parameters) need to be configured to support secure https (that is, the domain name of the above-mentioned artificial intelligence platform) access to the artificial intelligence platform, and at the same time configure the https domain name proxy / map to the http domain name of the annotation component (that is, the domain name of the above-mentioned annotation component), so as to achieve password-free secure access.

[0071] In summary, the method disclosed in the present invention realizes the transmission of the first access account information and the session identifier in the front end of the artificial intelligence platform through the first domain name proxy module in the front-end service of the artificial intelligence platform and the second domain name proxy module in the annotation component. The user does not need to enter the first access account information in the annotation component to log in to the annotation component without a password, thereby improving the user experience and data annotation efficiency.

[0072] Figure 8 A flow chart of a data annotation management method proposed in the present disclosure is further shown. Figure 1 The embodiment shown further explains step 103. Figure 8 The following steps may be included: Step 801, execute the first command, call the remote client component, and send the session identifier to the second domain name proxy module to log in to the annotation component.

[0073] In some embodiments, the artificial intelligence platform can execute a first command in a remote client component, thereby calling the remote client component and passing the session identifier from the port of the artificial intelligence platform to the port of the annotation component. When the second domain name proxy module in the annotation component listens to the port of the port of the annotation component and detects that the port has successfully received the session identifier, the session identifier can be sent to the remote server component of the annotation component, thereby using the remote server component to pass the session identifier to the login system of the annotation component, thereby bypassing the artificial intelligence platform and the front end of the annotation component, and realizing password-free login to the annotation component from the back-end microservice.

[0074] Step 802, in response to the first task query request of the creating user, obtain the network address of the annotation component at the time of login.

[0075] In some embodiments, when the creating user initiates a first task query request in the artificial intelligence platform, the artificial intelligence platform can obtain the network address of the login annotation component stored in the database by calling the database, etc. The present disclosure does not limit the network address when obtaining the login annotation component.

[0076] In some embodiments, the first task query request is used to request access to a details interface in a data annotation task.

[0077] In some embodiments, the network address when logging into the annotation component is the network address of the data annotation task.

[0078] Step 803: Execute a second command based on the network address to call the remote client component and access the details interface of the data annotation task in the annotation component.

[0079] In some embodiments, the artificial intelligence platform can execute a second command to call the remote client component to access the network address when logging into the annotation component, thereby opening the details interface of the data annotation task in the annotation component.

[0080] For example, in an optional embodiment, as Fig. 9 As shown, the jsonrpc component can be used to implement the artificial intelligence platform to perform operations in the annotation component, where the jsonrpc component in the artificial intelligence platform is a remote client component, and the jsonrpc component in the annotation component is a remote server component, where the port corresponding to the remote client component is port 32107, and the port corresponding to the annotation component is 8081. Then when the artificial intelligence platform executes the first command (i.e. Fig. 9 When the command 8199 / jsonrpc is shown in the figure to call the remote client component to request port 8199 corresponding to port 8081, the second domain name proxy service module nginx of the annotation component can send the session identifier requested by the artificial intelligence platform to the remote server component (i.e., the jsonrpc component in the annotation component) by listening to port 8081.

[0081] When the AI ​​platform executes the second command (i.e. Fig. 9 The command 8199 / project shown in the figure is used to call the remote client component to request port 8199 corresponding to port 8081. At this time, there is no need for the second domain name proxy service module of the annotation component to listen to port 8081. The artificial intelligence platform suffers from directly accessing the details interface of the data annotation task in the annotation component (i.e. Fig. 9Labelstudio interface shown).

[0082] In summary, the data annotation management method proposed in the present disclosure is applied to the artificial intelligence platform, including: executing a first command, calling a remote client component, sending a session identifier to a second domain name proxy module to log in to the annotation component; in response to the creation of a user's first task query request, obtaining the network address of the annotation component at the time of login; based on the network address, executing a second command to call the remote client component to access the details interface of the data annotation task in the annotation component. The method disclosed in the present disclosure, by utilizing a remote client component, enables the artificial intelligence platform to log in to the annotation component through the backend without a password, and can also realize the creation of a user's query on the annotation interface in the annotation component, thereby improving the user's operational convenience and data annotation efficiency.

[0083] Fig.10 A flow chart of a data annotation management method proposed in the present disclosure is further shown. Figure 1 The embodiment shown further explains step 105. Fig.10 The following steps may be included: Step 1001, based on the data labeling task request, determine the computing power node corresponding to the artificial intelligence platform for the data labeling task.

[0084] In some embodiments, the artificial intelligence platform can be communicated and connected with multiple computing power nodes. The intelligent scheduling system of the artificial intelligence platform can assign computing power nodes for user automatic labeling to the data labeling task based on information such as the amount of data to be labeled, the type of data to be labeled, the user level of the creating user, etc., and then label the data to be labeled by calling the inference model in the assigned computing power node, thereby realizing automatic labeling of the data to be labeled.

[0085] Step 1002, configure the second resource address of the computing power node for the data labeling task, so as to automatically label the data to be labeled in the data labeling task by using the reasoning model of the computing power node, the model interface of the reasoning model, the task address of the data labeling task, and the access information of the data labeling task.

[0086] In some embodiments, the reasoning model can be configured to the data labeling task through the model interface of the reasoning model, the task address of the data labeling task, and the access information of the data labeling task, so that the reasoning model can be used to automatically label the data to be labeled.

[0087] In some embodiments, the access information is generated by the annotation component when creating a data annotation task, and the access information is used to allow other devices and / or programs (such as artificial intelligence platforms, inference models, etc.) outside the annotation component to access the data annotation task.

[0088] In some embodiments, the second resource address may be a URL address of a computing power node.

[0089] In some embodiments, the URL address of the computing power node and the inference model in the computing power node can be configured into the data labeling task, so that after the computing power node is successfully connected to the labeling component, the labeling component can call the inference model in the computing power node to automatically label the data to be labeled.

[0090] In some embodiments, the labeling result of the automatically labeled data to be labeled can also be manually corrected by the creating user.

[0091] In some embodiments, the artificial intelligence platform can also train the reasoning model to improve the accuracy of automatic labeling.

[0092] Specifically, in response to a request to create a user's inference model training, a training node in the artificial intelligence platform for training the inference model can be determined; based on the training node, the inference model can be trained using a preset training data set and / or data that has been labeled in the data labeling task.

[0093] Among them, appropriate training nodes can be allocated to the inference model training request according to the user level of the creating user, the requirement for the accuracy of the inference model, etc.

[0094] In some embodiments, the training speeds of different training nodes and the CPU and / or GPU resources used for training may be the same or different.

[0095] In some embodiments, the CPU and / or GPU resources used by different computing nodes when automatically annotating the data to be annotated may be the same or different.

[0096] In some optional embodiments, the artificial intelligence platform also has an early warning function. Specifically, the artificial intelligence platform can determine the expected completion time of the data labeling task based on preset rules; when the execution time of the data labeling task exceeds the expected completion time, a timeout prompt is displayed in a second preset area of ​​the artificial intelligence platform, and / or the data labeling task is terminated, and the computing power node corresponding to the data labeling task is released.

[0097] In other words, in some optional embodiments, when the execution time of the data labeling task exceeds the expected completion time, a timeout prompt may be displayed only in the second preset area of ​​the artificial intelligence platform.

[0098] In some optional embodiments, when the execution time of a data labeling task exceeds the expected completion time, only the data labeling task may be terminated and the computing power node corresponding to the data labeling task may be released.

[0099] In some optional embodiments, when the execution time of a data labeling task exceeds the expected completion time, a timeout prompt can be displayed in a second preset area of ​​the artificial intelligence platform, the data labeling task is terminated, and the computing power node corresponding to the data labeling task is released.

[0100] For example, the artificial intelligence platform can manage the timeout of data labeling tasks through the intelligent scheduling system, and can perform timeout notification, error notification and other functions for data labeling tasks. The artificial intelligence platform can set timeouts for different types of tasks (training tasks, reasoning tasks, data labeling tasks, etc.). At the same time, the artificial intelligence platform can also set timeouts for different types of tasks of resources. For example, the timeout of the reasoning node is set to 1 day, and the number of notifications is 3 times. After the data labeling task runs for more than one day, a message notification will be made 3 times in the second preset area of ​​the artificial intelligence platform. At the same time, the action after the timeout can be set, such as setting the termination of the data labeling task after the timeout and releasing the resources occupied by the resource group.

[0101] Step 1003, obtaining the data annotation progress to display the annotation progress in the first preset area of ​​the artificial intelligence platform.

[0102] In some embodiments, the artificial intelligence platform can obtain the annotation progress of data in the data annotation task in the annotation component through an API interface, but not limited to this, and display the annotation progress in a first preset area of ​​the artificial intelligence platform to facilitate the creation of users to view the completion status of the data annotation task in real time.

[0103] In some embodiments, when the data to be labeled in the data labeling task is completed, the labeling component is used to export the labeling results of the data to be labeled in the data labeling task to the artificial intelligence platform in a preset file format.

[0104] In some embodiments, the preset file format is, for example, json, csv or other file formats, which is not limited in the present disclosure.

[0105] In summary, the data annotation management method proposed in the present disclosure is applied to an artificial intelligence platform, including: determining the computing power node for the data annotation task corresponding to the artificial intelligence platform based on the data annotation task request; configuring the second resource address of the computing power node for the data annotation task, so as to automatically annotate the data to be annotated in the data annotation task by using the reasoning model of the computing power node, the model interface of the reasoning model, the task address of the data annotation task, and the access information of the data annotation task; obtaining the annotation progress of the data, so as to display the annotation progress in the first preset area of ​​the artificial intelligence platform. The method disclosed in the present disclosure realizes the automatic annotation of the data to be annotated in the data annotation task by calling the computing power node corresponding to the artificial intelligence platform, thereby improving the data annotation efficiency.

[0106] Fig.11 A flow chart of a data annotation management method proposed in the present disclosure is further shown. Figure 1 The illustrated embodiment further explains, Fig.11 The following steps may be included: Step 1101, in response to a task sharing request of a creating user, determining a data annotation subtask corresponding to the shared user in the data annotation task, and generating second access account information of the annotation component corresponding to the shared user.

[0107] In some embodiments, the task sharing request is used to allocate part of the data to be labeled in the data labeling task to the sharing user.

[0108] In some embodiments, the task sharing request may also indicate a data annotation subtask assigned to the sharing user.

[0109] In some embodiments, the method for generating the second access account information is similar to the method for generating the first access account information, and reference may be made to the embodiment shown in step 101 and the description in related embodiments, which will not be repeated here.

[0110] Step 1102 , in response to the second task query request of the sharing user, based on the second access account information, log in to the annotation component to access the details interface of the data annotation subtask in the annotation component.

[0111] In some embodiments, upon receiving the second task query request from the shared user, the artificial intelligence platform can, based on the second access account information, log in to the annotation component of the front-end service or back-end service of the artificial intelligence platform without password (for details, see Figure 6 or Figure 8 The embodiment shown is not described in detail here) to access the details interface of the data annotation subtask in the annotation component.

[0112] In some embodiments, a shared user may only be able to access the detail interface of the data annotation subtask assigned to him or her, or may be able to access the detail interface of the entire data annotation task, which will not be described in detail here.

[0113] For example, Fig.12As shown, after the user in the artificial intelligence platform creates a data annotation task, the data annotation task can be shared individually or in batches with other users of the artificial intelligence platform. After successful sharing, other users can view the annotation task, and the user can also reclaim the permission of the data annotation task. After reclaiming, other users cannot view the data annotation task; when sharing data annotation tasks, the user needs to set the account password of the shared user to access the annotation component, as well as access internal interface information (i.e. the second access account information mentioned above), and needs to call the API of the annotation component to add users. After adding the user's account, the shared user can use the account information to perform single sign-on to the annotation component interface of the shared annotation task; collaborative data annotation requires the use of the system annotation function in the annotation component to assign projects to shared users, and at the same time set the annotation content in the annotation component for different users, so that shared users can receive their own data annotation task work.

[0114] In summary, the data annotation management method proposed in the present disclosure is applied to an artificial intelligence platform, including: responding to a task sharing request of a creating user, determining a data annotation subtask corresponding to a shared user in a data annotation task, and generating a second access account information of an annotation component corresponding to the shared user; responding to a second task query request of a shared user, based on the second access account information, logging into the annotation component to access the details interface of the data annotation subtask in the annotation component. The method of the present disclosure allocates data annotation subtasks and second access account information to a shared user by creating a task sharing request of a user, thereby realizing a password-free login of a shared user to an annotation component, as well as sharing and allocating data annotation tasks, providing users with a more flexible annotation method, and improving the convenience of user operation and data annotation efficiency.

[0115] In order to implement the data annotation management method provided by the embodiment of the present disclosure, the embodiment of the present disclosure also provides a data annotation management device, such as Fig.13 As shown, the data annotation management device 1300 includes: A determining unit 1301 is configured to determine first access account information of the annotation component in response to a request to create a data annotation task of a user; The acquiring unit 1302 is configured to acquire a session identifier of the annotation component based on the first access account information; A login unit 1303, configured to log in to the annotation component based on the session identifier of the annotation component; A creating unit 1304, configured to create a data annotation task using an annotation component based on the data annotation task request; A synchronization unit 1305 is used to synchronize the data to be annotated in the artificial intelligence platform to the data annotation task based on the access interface of the annotation component; The labeling unit 1306 is used to automatically label the data to be labeled in the data labeling task.

[0116] In some embodiments, the acquisition unit 1302 is also used to access the first resource address of the annotation component based on the first access account information; when the first resource address is successfully accessed, receive the cross-site request information sent by the annotation component; based on the first access account information and the cross-site request information, obtain the session identifier of the annotation component.

[0117] In some embodiments, the login unit 1303 is also used to log in to the annotation component from the front-end service of the artificial intelligence platform using the first domain name proxy module based on the session identifier, and the first domain name proxy module corresponds to the front-end service of the artificial intelligence platform; and / or, based on the session identifier, execute the first command to call the remote server component to log in to the annotation component from the back-end service of the artificial intelligence platform, and the remote server component corresponds to the back-end service of the artificial intelligence platform.

[0118] In some embodiments, the login unit 1303 is also used to use the first domain name proxy module to map the domain name of the artificial intelligence platform to the domain name address of the second domain name proxy module of the annotation component; based on the domain name address, the first access account information and the session identifier are sent to the annotation component to log in to the annotation component.

[0119] In some embodiments, the login unit 1303 is further used to execute the first command, call the remote server component, and send the session identifier to the second domain name proxy module to log in to the annotation component.

[0120] In some embodiments, the login unit 1303 is also used to respond to the first task query request of the creating user and obtain the network address of the annotation component at the time of login; based on the network address, execute the second command to call the remote server component and access the details interface of the data annotation task in the annotation component.

[0121] In some embodiments, the creation unit 1304 is also used to determine the processor resources used to create the data labeling task based on the candidate resource group corresponding to the creating user, where the candidate resource group belongs to the artificial intelligence platform; based on the processor resources and the data path of the data to be labeled corresponding to the data labeling task request, create the data labeling task in the labeling component.

[0122] In some embodiments, the labeling unit 1306 is also used to determine the computing power node for the data labeling task corresponding to the artificial intelligence platform based on the data labeling task request; configure the second resource address of the computing power node for the data labeling task, so as to use the inference model of the computing power node, the model interface of the inference model, the task address of the data labeling task, and the access information of the data labeling task to automatically label the data to be labeled in the data labeling task; obtain the labeling progress of the data to display the labeling progress in the first preset area of ​​the artificial intelligence platform.

[0123] In some embodiments, the labeling unit 1306 is also used to determine the training nodes in the artificial intelligence platform for training the reasoning model in response to the reasoning model training request created by the user; based on the training nodes, the reasoning model is trained using the preset training data set and / or the data that has been labeled in the data labeling task.

[0124] In some embodiments, the labeling unit 1306 is also used to determine the expected completion time of the data labeling task based on preset rules; when the execution time of the data labeling task exceeds the expected completion time, a timeout prompt is displayed in a second preset area of ​​the artificial intelligence platform, and / or the data labeling task is terminated, and the computing power node corresponding to the data labeling task is released.

[0125] In some embodiments, the creation unit 1304 is further used to respond to the task sharing request of the creating user, determine the data annotation subtask corresponding to the shared user in the data annotation task, and generate the second access account information of the annotation component corresponding to the shared user; respond to the second task query request of the shared user, log in to the annotation component based on the second access account information to access the details interface of the data annotation subtask in the annotation component. In some embodiments, the first access account information includes at least: the login name of the creating user, the login password of the creating user, and the information of the access interface between the artificial intelligence platform and the annotation component.

[0126] In some embodiments, the labeling unit 1306 is also used to use the labeling component to export the labeling results of the data to be labeled in the data labeling task to the artificial intelligence platform in a preset file format when the data to be labeled in the data labeling task is completed.

[0127] It should be noted that: the data annotation management device provided in the above embodiment only uses the division of the above program modules as an example when performing data annotation management. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the data annotation management device is divided into different program modules to complete all or part of the processing described above. In addition, the data annotation management device provided in the above embodiment and the data annotation management method embodiment provided in the embodiment of the present disclosure belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0128] Fig.14 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure is shown in FIG. Fig.14 As shown, the electronic device 1400 includes at least one processor 1402; and a memory 1401 communicatively connected to the at least one processor 1402; wherein the memory 1401 stores commands that can be executed by the at least one processor 1402, and the commands are executed by the at least one processor 1402 to implement the steps of the data labeling management method described in the embodiment of the present disclosure.

[0129] Optionally, the electronic device may specifically be a data labeling management device of an embodiment of the present application, and the electronic device may implement the corresponding processes implemented by the data labeling management device in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0130] It is understood that the electronic device also includes a communication interface 1403. The various components in the electronic device are coupled together through a bus system 1404. It is understood that the bus system 1404 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1404 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.14 Various buses are labeled as bus system 1404.

[0131] It can be understood that the memory 1401 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. The present method is described by way of example but not limitation, and many forms of RAM may be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 1401 described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0132] The method disclosed in the above embodiment of the present disclosure can be applied to the processor 1402, or implemented by the processor 1402. The processor 1402 has the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 1402 or the command in the form of software. The above processor 1402 can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1402 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 1401, and the processor 1402 reads the information in the memory 1401 and completes the steps of the above method in combination with its hardware.

[0133] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general purpose processors, controllers, MCUs, microprocessors, or other electronic components to execute the aforementioned method.

[0134] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium storing computer commands, wherein the computer commands are used to enable the computer to implement the steps of the data annotation management method described in the embodiment of the present disclosure when executed.

[0135] The embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the steps of the data annotation management method described in the embodiment of the present disclosure when executed by a processor.

[0136] Optionally, the computer-readable storage medium can be applied to the data labeling management device in the embodiments of the present application, and the computer command enables the computer to execute the corresponding processes implemented by the data labeling management device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0137] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0138] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0139] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0140] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program commands, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.

[0141] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0142] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data annotation management method, characterized in that: The method is applied to an artificial intelligence platform, and the method comprises: In response to creating a data annotation task request of a user, determining first access account information of the annotation component; Based on the first access account information, obtaining a session identifier of the annotation component; Based on the session identifier of the annotation component, logging into the annotation component; Based on the data annotation task request, create a data annotation task using the annotation component; Based on the access interface of the annotation component, the data to be annotated in the artificial intelligence platform is synchronized to the data annotation task; Automatically label the data to be labeled in the data labeling task.

2. The method according to claim 1, characterized in that The acquiring, based on the first access account information, a session identifier of the annotation component comprises: Accessing a first resource address of the annotation component based on the first access account information; When the first resource address is successfully accessed, receiving cross-site request information sent by the annotation component; Based on the first access account information and the cross-site request information, a session identifier of the annotation component is obtained.

3. The method according to claim 1, characterized in that: The logging into the annotation component based on the session identifier of the annotation component comprises: Based on the session identifier, using a first domain name proxy module, logging into the annotation component from a front-end service of the artificial intelligence platform, the first domain name proxy module corresponding to the front-end service of the artificial intelligence platform; and / or, Based on the session identifier, a first command is executed to call a remote client component to log in to the annotation component from a backend service of the artificial intelligence platform, and the remote client component corresponds to the backend service of the artificial intelligence platform.

4. The method according to claim 3, characterized in that The step of logging into the annotation component from the front-end service of the artificial intelligence platform based on the session identifier and using the first domain name proxy module includes: Using the first domain name proxy module, mapping the domain name of the artificial intelligence platform to the domain name address of the second domain name proxy module of the annotation component; Based on the domain name address, the first access account information and the session identifier are sent to the annotation component to log in to the annotation component.

5. The method according to claim 3, characterized in that: The executing a first command based on the session identifier to call a remote client component and log in to the annotation component from the backend service of the artificial intelligence platform includes: The first command is executed to call the remote client component and send the session identifier to the second domain name proxy module to log in to the annotation component.

6. The method according to claim 5, characterized in that The method further comprises: In response to the first task query request of the creating user, obtaining the network address of the annotation component at the time of login; Based on the network address, a second command is executed to call the remote client component to access the details interface of the data annotation task in the annotation component.

7. The method according to claim 1, characterized in that The creating a data annotation task by using the annotation component based on the data annotation task request includes: Determining processor resources for creating the data annotation task based on a candidate resource group corresponding to the creating user, wherein the candidate resource group belongs to the artificial intelligence platform; Based on the processor resources and the data path of the to-be-annotated data corresponding to the data annotation task request, the data annotation task is created in the annotation component.

8. The method according to claim 1, characterized in that The automatically labeling the data to be labeled in the data labeling task includes: Based on the data labeling task request, determining a computing power node corresponding to the artificial intelligence platform for the data labeling task; Configuring a second resource address of the computing power node for the data labeling task, so as to automatically label the data to be labeled in the data labeling task by using the reasoning model of the computing power node, the model interface of the reasoning model, the task address of the data labeling task, and the access information of the data labeling task; Obtain the labeling progress of the data to display the labeling progress in a first preset area of ​​the artificial intelligence platform.

9. The method according to claim 8, characterized in that The method further comprises: In response to the inference model training request of the creating user, determining a training node in the artificial intelligence platform for training the inference model; Based on the training node, the inference model is trained using a preset training data set and / or the data that has been labeled in the data labeling task.

10. The method according to claim 1, characterized in that The method further comprises: Based on preset rules, determine the estimated completion time of the data labeling task; When the execution time of the data labeling task exceeds the expected completion time, a timeout prompt is displayed in a second preset area of ​​the artificial intelligence platform, and / or the data labeling task is terminated, and the computing power node corresponding to the data labeling task is released.

11. The method according to claim 1, characterized in that: The method further comprises: In response to the task sharing request of the creating user, determining the data annotation subtask corresponding to the shared user in the data annotation task, and generating second access account information of the annotation component corresponding to the shared user; In response to the second task query request of the sharing user, based on the second access account information, the annotation component is logged in to access the detail interface of the data annotation subtask in the annotation component.

12. The method according to claim 1, characterized in that The first access account information includes at least: the login name of the creating user, the login password of the creating user, and information about the access interface between the artificial intelligence platform and the annotation component.

13. The method according to claim 1, characterized in that The method further comprises: When the data to be labeled in the data labeling task is labeled, the labeling component is used to export the labeling results of the data to be labeled in the data labeling task to the artificial intelligence platform in a preset file format.

14. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the data labeling management method according to any one of claims 1-13.

15. A non-transitory computer-readable storage medium storing computer commands, characterized in that: The computer command is used to enable the computer to execute the data annotation management method according to any one of claims 1-13.

Images