Data set labeling method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data set labeling method, electronic equipment, a storage medium and a program product. The method comprises the steps of deploying an open source data set labeling model to a preset container cluster and / or deploying a preset artificial intelligence model to the preset container cluster according to model modification parameters; the effective data set is uploaded to object storage for segmentation and encryption storage, and a to-be-labeled data set is obtained; according to the labeling task information, when it is detected that a labeling mode is model labeling, calling a to-be-labeled data set according to the labeling task information, and calling a preset artificial intelligence model and / or an open source data labeling model from a preset container cluster as a target labeling model; decrypting the data set to be labeled; and inputting the decrypted data set into the target labeling model to generate labeled data, and storing the labeled data into the object storage. According to the method, automation and configurability of the labeling process are achieved, and high flexibility and safety are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data set labeling method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] Data labeling is the conversion of raw data into structured information that can be used by algorithms, and is applied to training set construction, validation and test set production, and quality review. For various data forms such as images, texts, speech and videos, the labeling results are directly related to the consistency of subsequent algorithm training and analysis.

[0003] The prior art usually adopts a manual labeling form, and through task splitting and distribution, the labeling personnel views samples one by one in a tool interface and enters or revises the labels; in important tasks, multiple rounds of cross review and sampling inspection are usually set, and the labeling and review under different data types and task requirements are completed by relying on personnel experience to meet the established standards.

[0004] However, as the data size and the number of concurrent tasks increase, processing one by one occupies a large amount of manpower and time, and the manual labeling steps are many, so that the data labeling efficiency is low, and it is difficult to meet the production demand. SUMMARY

[0005] The embodiments of the present application provide a data set labeling method, an electronic device, a storage medium and a program product, so as to achieve labeling automation and improve the data set labeling efficiency.

[0006] In a first aspect, the embodiments of the present application provide a data set labeling method, comprising:

[0007] Obtaining model modification parameters of a preset artificial intelligence model modified by a user and / or an open source data set labeling model uploaded by the user;

[0008] Deploying the open source data set labeling model to a preset container cluster and / or deploying the preset artificial intelligence model to the preset container cluster according to the model modification parameters;

[0009] Obtaining an original data set, performing data preprocessing on the original data set, and obtaining an effective data set;

[0010] Uploading the effective data set to an object storage for split encryption storage, and obtaining a to-be-labeled data set stored in the object storage;

[0011] Obtaining labeling task information, and determining a labeling mode according to the labeling task information;

[0012] When it is detected that the labeling mode is model labeling, a data set to be labeled is called from the object storage according to the labeling task information, and the preset artificial intelligence model and / or the open source data labeling model are called as target labeling models from the preset container cluster according to the labeling task information;

[0013] The data set to be labeled is decrypted to obtain a decrypted data set;

[0014] The decrypted data set is input into the target labeling model to generate labeled data, and the labeled data is stored in the object storage.

[0015] In a possible implementation, the preset container cluster is a public container cluster or a private container cluster, and the preset container cluster is deployed with a distributed scheduling cluster, and the distributed scheduling cluster includes one control node and at least one worker node;

[0016] The method further includes:

[0017] When it is detected that the preset container cluster is a public container cluster, the preset artificial intelligence model is deployed to each worker node in the distributed scheduling cluster according to the model modification parameter, so as to realize distributed deployment of the preset artificial intelligence model to the public container cluster;

[0018] And / or when it is detected that the preset container cluster is a private container cluster, the open source data set labeling model is deployed to each worker node in the distributed scheduling cluster, so as to realize distributed deployment of the open source data set labeling model to the private container cluster.

[0019] In a possible implementation, the method further includes:

[0020] Obtaining a model address of the open source data set labeling model;

[0021] Loading the open source data set labeling model into each worker node in the distributed scheduling cluster according to the model address, so as to realize deployment of the open source data set labeling model to each worker node in the distributed scheduling cluster.

[0022] In a possible implementation, the method further includes:

[0023] upload the valid data set to an object storage to split the valid data set to obtain a plurality of groups of to-be-encrypted data after encryption;

[0024] encrypt each to-be-encrypted data to obtain encrypted data, and store each encrypted data in the object storage in the form of an encrypted file to obtain a to-be-labeled data set stored in the object storage.

[0025] In a possible implementation, the data preprocessing on the original data set to obtain the valid data set comprises:

[0026] performing data deduplication processing on the original data set to obtain a first intermediate data set;

[0027] performing data desensitization processing on the first intermediate data set to obtain a second intermediate data set;

[0028] performing data detoxification processing on the second intermediate data set to obtain a third intermediate data set;

[0029] performing data enhancement processing on the third intermediate data set to obtain the valid data set.

[0030] In a possible implementation, the original data set comprises text data, image data, audio data and video data.

[0031] The data preprocessing on the original data set further comprises:

[0032] removing non-text data or specific characters from the text data to obtain valid text data;

[0033] performing image quality adjustment processing on the image data to obtain valid image data;

[0034] performing noise reduction processing on the audio data to obtain valid audio data;

[0035] performing video quality adjustment processing on the video data to obtain valid video data.

[0036] In a possible implementation, after the decrypted data set is input into the target labeling model to generate labeled data, the method further comprises:

[0037] performing display processing on the labeled data to obtain labeling adjustment operations performed by a user in response to the labeled data, to obtain adjusted labeled data;

[0038] storing the adjusted labeled data in the object storage.

[0039] In a second aspect, an embodiment of the present application provides a data set labeling apparatus, comprising:

[0040] an information obtaining module, configured to obtain model modification parameters of a preset artificial intelligence model modified by a user and / or an open source dataset labeling model uploaded by the user;

[0041] a deployment execution module, configured to deploy the open source dataset labeling model to a preset container cluster and / or deploy the preset artificial intelligence model to the preset container cluster according to the model modification parameters;

[0042] an information processing module, configured to obtain an original dataset, and perform data preprocessing on the original dataset to obtain an effective dataset;

[0043] The information processing module is further configured to upload the effective dataset to an object storage for split encryption storage to obtain a to-be-labeled dataset stored in the object storage.

[0044] The information processing module is further configured to obtain labeling task information, and determine a labeling manner according to the labeling task information.

[0045] The labeling execution module is configured to, when it is detected that the labeling manner is model labeling, call a to-be-labeled dataset from the object storage according to the labeling task information, and call the preset artificial intelligence model and / or the open source data labeling model as a target labeling model from the preset container cluster according to the labeling task information.

[0046] The labeling execution module is further configured to decrypt the to-be-labeled dataset to obtain a decrypted dataset.

[0047] The labeling execution module is further configured to input the decrypted dataset into the target labeling model to generate labeled data, and store the labeled data in the object storage.

[0048] In a third aspect, an electronic device is provided, including a memory and a processor.

[0049] The memory stores computer execution instructions.

[0050] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0051] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer execution instructions. The computer execution instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0052] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program which, when executed by a processor, implements the first aspect and / or various possible implementation manners of the first aspect.

[0053] The data set labeling method, the electronic device, the storage medium and the program product provided by the embodiments of the present application realize the explicitness and controllability of the model source and the parameter configuration by obtaining the model modification parameter of the preset artificial intelligence model modified by the user and / or the open source data set labeling model uploaded by the user, provide the referable model entry and the parameter entry for subsequent deployment and calling, and enable the model to be flexibly selected according to the task requirement. The standardization online and versioned operation of the model are realized by deploying the open source data set labeling model to the preset container cluster and / or deploying the preset artificial intelligence model to the preset container cluster according to the model modification parameter, and it is ensured that the target model can be obtained in the labeling stage. The consistency and standardization of the input data are realized by obtaining the original data set and performing data preprocessing to form an effective data set, and it is ensured that the data set meeting the processing requirement is received in the subsequent process. The security protection and segmented management of the large amount of data are realized by uploading the effective data set to the object storage and performing cutting and encryption storage to form the data set to be labeled stored in the object storage, and the security and traceability of the data are enhanced.

[0054] The clear mapping of the task configuration to the execution path is realized by obtaining the labeling task information and determining the labeling mode according to the labeling task information, so that the subsequent action can be performed in the pre-set mode. When the labeling mode is model labeling, the data set to be labeled is called from the object storage according to the labeling task information, and the preset artificial intelligence model and / or the open source data set labeling model are called from the preset container cluster as the target labeling model, so that the data side and the model side are synchronously connected, and the connection efficiency from the task triggering to the execution is improved. The encrypted data set is obtained by decrypting the data set to be labeled, the safe switching between the encrypted storage and the plaintext calculation is realized, and it is ensured that the data input to the target labeling model is complete, readable and consistent with the original content. Finally, the decrypted data set is input to the target labeling model to generate the labeled data, and the labeled data is stored in the object storage, the closed link from the output to the storage is realized, the subsequent referability of the labeling result is enhanced, and the overall labeling efficiency is improved to meet the production requirement. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application.

[0056] Figure 1 A flowchart of the data set labeling method provided by the present application is shown in the figure;

[0057] Figure 2 A method flow diagram for data preprocessing of a raw data set provided for an embodiment of the present application is shown in FIG. 1.

[0058] Figure 3 A method flow diagram for adjusting labeled data provided for an embodiment of the present application is shown in FIG. 2.

[0059] Figure 4 A structure diagram of a data set labeling device provided for the present application is shown in FIG. 3.

[0060] Figure 5 A structure diagram of an electronic device provided for the present application is shown in FIG. 4.

[0061] The specific embodiments of the present application have been shown and described in the above-described drawings and text. These drawings and text are not meant to limit the scope of the present application in any way, but merely to illustrate the concept of the present application to one of ordinary skill in the art by reference to a particular embodiment. DETAILED DESCRIPTION

[0062] The exemplary embodiments will be described in detail herein below with reference to the drawings. The following description is merely exemplary in nature and is not intended to limit the scope of the present application, as described throughout this document. Rather, the following description is intended only to illustrate the exemplary embodiments, which will be best understood by reference to the drawings, wherein like reference numerals refer to like parts throughout the various drawings.

[0063] The inventive concept of the present application is to provide a data set labeling method to improve the efficiency and accuracy of data labeling. Specifically, on the model side: on the one hand, the model modification parameters of the pre-set artificial intelligence model modified by the user are received, and on the other hand, the open source data set labeling model uploaded by the user is received, on the basis of which the open source data set labeling model is deployed to the preset container cluster, and / or the pre-set artificial intelligence model is deployed to the preset container cluster according to the model modification parameters, so that the subsequent labeling stage can directly call the online target model instance. On the data side: the original data set is obtained and pre-processed to obtain an effective data set; then the effective data set is uploaded to the object storage and stored after being cut and encrypted, forming a to-be-labeled data set stored in the object storage, ensuring that the subsequent reading based on the labeling task information is faced with a data object with unified format, positioning and decryption. On the execution side: the labeling task information is obtained, and the labeling method is determined accordingly; when it is detected that the labeling method is model labeling, the to-be-labeled data set is called from the object storage according to the labeling task information, and the pre-set artificial intelligence model and / or the open source data set labeling model are called as the target labeling model from the preset container cluster according to the labeling task information; after the to-be-labeled data set is decrypted, the decrypted data set is input into the target labeling model to generate labeled data, and the labeled data is stored in the object storage. Therefore, the model side, the data side and the task side are connected through the labeling task information, realizing batch processing and stable output under high concurrency samples, and solving the problem of slow artificial checking processing efficiency.

[0064] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0065] Figure 1 The flowchart of the data set labeling method provided by the present application is shown in FIG. 1, which comprises the following steps. Figure 1

[0066] S11, obtaining the model modification parameters of the pre-set artificial intelligence model modified by the user and / or the open source data set labeling model uploaded by the user.

[0067] ​In the embodiment, the model modification parameter is a configurable parameter set of the preset artificial intelligence model, including but not limited to weight file identification, structure switch, analysis threshold, input and output size, pre-processing and post-processing rule, version information and integrity check value. The open source data set annotation model is a model provided by a user for data set annotation and its description information, including model file identification, compatible input and output preset information, version information and integrity check value. By receiving the above two types of information at the input interface, the subsequent deployment and calling object is uniformly determined and recorded.

[0068] In the manner of obtaining the model modification parameter, first, the model identification information of the preset artificial intelligence model is received to locate the target preset artificial intelligence model; then, the model modification parameter corresponding to the model is received, the numerical interval, enumeration value and format are checked according to the parameter definition, and the consistency check is performed on the pre-processing, post-processing and input and output size type parameters, and the parameter set passing the check is generated; the parameter set passing the check is stored in association with the corresponding model identification information, forming a traceable parameter version record. By checking the integrity and consistency of the model modification parameter, the determination and version management of the parameter configuration are realized.

[0069] In the manner of obtaining the open source data set annotation model, the model file uploaded by the user and its associated information are received, the integrity check value and version information of the model file are read, the file format and input and output preset information are checked, and the model identification information and compatibility information are registered in combination with the model description information; the registered record of the model passing the check is generated and associated with the user identity information, forming a model entry that can be used for subsequent deployment and calling. By registering and checking the open source data set annotation model, the access and availability of the external model are realized.

[0070] In the case where the two manners coexist, the target model entry or parameter entry that can be referenced is generated after the record is completed, and the content at least includes model identification information, version information, integrity check value, input and output agreement and resource requirement summary, and the entry is connected to the deployment and calling interface of the subsequent step. By uniformly registering the target model entry and the parameter entry, the consistent reference and traceability of the model and its parameters by the subsequent process are realized.

[0071] S12, deploying the open source data set annotation model to a preset container cluster and / or deploying the preset artificial intelligence model to the preset container cluster according to the model modification parameter.

[0072] In the embodiment, a model service instance that can be subsequently invoked is established in a preset container cluster. The preset container cluster is a containerized running environment for running the model service, providing computing resource allocation, instance scheduling, service entry registration, etc. When deploying the open source dataset annotation model, the model file and its description information are imported to generate a corresponding containerized running package and start configuration, which at least includes the model file location, start command, input and output convention, service port and resource quota; the running package is registered to the container image repository, and a deployment manifest is created in the preset container cluster, which includes the number of replicas, computing and storage resource requests, environment variables, key and certificate references, and external access ports. Through standardized running package and deployment manifest, reusable deployment and stable running of the open source dataset annotation model in the preset container cluster are realized.

[0073] When deploying the preset artificial intelligence model according to the model modification parameter, first, the baseline configuration of the preset artificial intelligence model and the corresponding containerized running package are read, and a new start configuration is generated according to the model modification parameter, such as weight file binding, input and output size, concurrency, etc.; the new start configuration and the running package are associated to generate a corresponding deployment manifest and version identifier; a model service instance is created or updated in the preset container cluster according to the version identifier, and the new and old instances are switched in a step-by-step replacement manner without interrupting the service, and a rollback point and configuration verification summary are registered for the version; after starting and probe verification, a callable service identifier and access entry are allocated. Through the application and versioned deployment of the model modification parameter, the preset artificial intelligence model is realized in the preset container cluster. In summary, by deploying the open source dataset annotation model to the preset container cluster and / or deploying the preset artificial intelligence model to the preset container cluster according to the model modification parameter, the standardized online and versioned running of the model are realized, and it is ensured that the target model can be obtained in the annotation stage.

[0074] S13, obtaining an original dataset, performing data preprocessing on the original dataset to obtain an effective dataset.

[0075] In this embodiment, the original data set refers to the data set provided or collected by the user before the start of the labeling task. Data preprocessing is a cleaning, standardization and structure unification process for the original data set, aiming to remove redundancy and eliminate irregular content without changing the business meaning. The effective data set refers to the data set that meets the subsequent split encryption storage and model calling input specification after data preprocessing. Further, for cross-source data, the correlation is established according to time or primary key alignment to ensure that the subsequent on-demand extraction can be output in a stable order. The integrity of the preprocessed set is checked, and the effective data set is bound with its version information as the input of split encryption storage. In summary, by obtaining the original data set and performing data preprocessing, the effective data set is formed, realizing the consistency and standardization of the input data, and ensuring that the subsequent process receives a data set that meets the processing requirements.

[0076] S14, upload the effective data set to the object storage for split encryption storage, and obtain the data set to be labeled stored in the object storage.

[0077] In this embodiment, the effective data set is divided into pieces according to the preset split rule. The split rule at least includes the size parameter of the split and the sequence parameter of the split, which is used to form a continuous indexable piece sequence on the object storage side. Each piece after splitting is corresponded to its batch identification and sequence number, so that it can be recovered in sequence for subsequent reading. By splitting the effective data set, multiple groups of data to be encrypted are formed, which facilitates stable transmission and segmented storage of large files. In the encryption and storage process, each data to be encrypted is executed for encryption processing, which at least includes selecting an encryption method and generating a key parameter required for encryption, and writing the encrypted data as an encrypted file into the object storage. To ensure subsequent positioning and decryption, a record item is generated for each encrypted file, which at least includes batch identification, piece sequence number, encryption method identification and verification information. After completing the writing, the storage positioning information generated on the object storage side is returned and bound with the batch. By executing encryption on each piece and writing it into the object storage as an encrypted file, the safekeeping and segmented management of large volume data are realized, and the confidentiality and traceability of the data are enhanced.

[0078] S15, obtain the labeling task information, and determine the labeling method according to the labeling task information.

[0079] In the embodiment, the labeling task information is a set of configuration information submitted by the user through the WEB front end, such as the identification of the target data set, the target data type, the expected label definition, the labeling range, and the labeling method field; the labeling task information can also include selection items and description fields related to the model. After receiving the labeling task information, the field integrity check and format check are completed, the information is associated with the unique task number for storage for subsequent use. By registering the labeling task information, the data object and configuration boundary corresponding to this job are determined. When determining the labeling method, the labeling method field in the labeling task information can be read, and the value of the field is taken as the execution method record of this job. If a single labeling method is given in the labeling task information, the method is taken as the execution path of the subsequent link; if a combination of multiple labeling methods is given in the labeling task information, the combination is recorded for subsequent processing according to the recorded combination. The labeling method field in the labeling task information is registered to generate a method determination result that can be referenced by subsequent steps. In summary, by obtaining the labeling task information and determining the labeling method according to the labeling task information, a clear mapping from task configuration to execution path is realized, so that subsequent actions can be performed according to the preset method.

[0080] S16, when it is detected that the labeling method is model labeling, the labeling task information is used to call the data set to be labeled from the object storage, and the preset artificial intelligence model and / or open source data labeling model are called from the preset container cluster as the target labeling model according to the labeling task information.

[0081] In the embodiment, the labeling task information includes the positioning information of the data set to be labeled in the object storage and the identification information of the model to be called. When the labeling method is model labeling, the object storage positioning information in the labeling task information is read to obtain the data set to be labeled corresponding to the task, which is used for subsequent decryption and input. By positioning the object storage entry according to the labeling task information, the calling preparation of the data set to be labeled is completed and the data range and sequence are ensured to be consistent with the task. Then, the model identification information in the labeling task information is read, and the preset artificial intelligence model instance and / or open source data set labeling model instance corresponding to the identification are searched in the preset container cluster to obtain the service identification and access portal that can be called. In summary, the data set to be labeled is called from the object storage according to the labeling task information, and the preset artificial intelligence model and / or open source data set labeling model are called from the preset container cluster as the target labeling model, which realizes the synchronous docking of the data side and the model side, and improves the connection efficiency from task triggering to execution.

[0082] S17, the data set to be labeled is decrypted to obtain a decrypted data set.

[0083] In this embodiment, according to the encrypted file set corresponding to the current task, the decryption processing is sequentially performed in the order of the formed fragments when uploading, and each fragment is restored to a readable original data format. By decrypting each encrypted file and keeping the fragment order consistent, a decrypted data set corresponding to the data set to be labeled is obtained. When all fragments are decrypted, the fragments are combined into a file set consistent with the effective data set in the order of the fragments. Further, the data set to be labeled can be encrypted in a symmetric encryption or asymmetric encryption manner. To realize data decryption, the decryption key corresponding to the key identification information carried in the labeling task information is obtained from the secure key management module. The key management module can be a locally deployed key management system or a key storage service based on cloud services. In summary, by decrypting the data set to be labeled and obtaining the decrypted data set, the secure switching between ciphertext storage and plaintext calculation is realized, ensuring that the data input to the target labeling model is complete, readable and consistent with the original content.

[0084] S18, input the decrypted data set into the target labeling model, generate labeled data, and store the labeled data in the object storage.

[0085] In this embodiment, the decrypted data set is input into the target labeling model to generate labeled data and complete storage. The decrypted data set is checked for size, encoding, and channel number consistency according to the input requirements of the target labeling model; after the check, the decrypted data set is input into the target labeling model to obtain a result set of labeled data. The labeled data is associated with its source identification to generate a searchable record item; the labeled data is organized according to the preset data format to form a file or record for storage and stored in the object storage. Further, the target labeling model can be a machine learning or deep learning model of various types, including but not limited to a text labeling model based on natural language processing, a target detection model based on image recognition, or a multi-modal labeling model based on audio and video analysis. The target labeling model can be pre-trained or fine-tuned on specific field data to adapt to the needs of the current labeling task. By realizing automatic labeling of the decrypted data set and standardized storage of the labeled data, a closed link from output to storage is realized, the subsequent referenceability of the labeling results is enhanced, and the data labeling efficiency is improved to meet production needs.

[0086] In one embodiment, the preset container cluster is a public container cluster or a dedicated container cluster, and the preset container cluster is deployed with a distributed scheduling cluster, and the distributed scheduling cluster includes one control node and at least one worker node.

[0087] In the embodiment, the preset container cluster is a Kubernetes cluster (K8S cluster), that is, a container running environment that is scalable, uniformly scheduled, and resource-isolated, which is built based on a Kubernetes container orchestration system. The K8S cluster can be preset as a public container cluster or a dedicated container cluster. The public container cluster refers to a K8S cluster environment that can be shared by multiple users or projects. The dedicated container cluster is an isolated running environment dedicated to a single project or user. A Ray cluster is deployed in the preset container cluster as a distributed scheduling cluster. The Ray cluster is composed of a control node and at least one worker node. The control node is used to accept and schedule distributed labeling tasks submitted by users, and the worker node actually carries out the task execution process, including data reading, labeling model analysis, result collection and feedback, and the like. Each worker node is controlled by the resource scheduling of K8S, and supports automatic elastic scaling.

[0088] The K8S cluster is mainly used for the orchestration, running and lifecycle management of containers. By deploying different types of task instances to suitable computing nodes, the overall availability and management efficiency are improved. The Ray cluster is used for distributed scheduling and state tracking of tasks, realizes parallel execution and dynamic allocation of complex labeling tasks, and improves the overall execution efficiency. At the same time, RayServe is integrated into the Ray cluster as a unified external service interface for interfacing with the upper task management platform or external triggering module, realizing a unified access portal for task triggering, state querying, result feedback and other services, and simplifying the integration complexity. Through the cooperation of the cluster architecture of the embodiment, resource elastic scheduling, efficient concurrent task execution and unified service export are realized, and the processing capacity and scalability of the labeling platform in a multi-task high-concurrency scenario are improved.

[0089] Next, an implementation manner of deploying the model in step S12 is provided. Based on the above embodiment, the following is included:

[0090] S121, when it is detected that the preset container cluster is a public container cluster, the pre-stored artificial intelligence model is deployed to each worker node in the distributed scheduling cluster according to the model modification parameter, so as to realize distributed deployment of the pre-stored artificial intelligence model to the public container cluster;

[0091] S122, when it is detected that the preset container cluster is a dedicated container cluster, the open source dataset labeling model is deployed to each worker node in the distributed scheduling cluster, so as to realize distributed deployment of the open source dataset labeling model to the dedicated container cluster.

[0092] In the embodiment, when the preset container cluster is a public container cluster, a preset artificial intelligence model is selected for deployment according to a current task configuration. The preset artificial intelligence model usually refers to a general labeling model, such as an image target detection model, a semantic segmentation model or a text classification model, and has strong generalization ability and low deployment threshold. Through a model parameter optimization interface, runtime parameters of the model (such as batch size, thread number, GPU usage strategy and the like) are adjusted to a mode suitable for current public cluster resource utilization, and then the model image is uniformly distributed to the Pod containers corresponding to each worker node, and the Ray cluster Actor mechanism is used to complete model instantiation and initialization, so as to realize parallel deployment on each worker node. In the deployment process, the K8S ConfigMap and Volume mechanisms can be combined to mount the configuration files and parameter directories required by the model, to ensure the consistency of model initialization. Through distributed deployment, the model service capability is improved, and the stability and fairness of model execution can be ensured in a multi-tenant shared environment.

[0093] When the preset container cluster is a dedicated container cluster, a differentiated deployment strategy is adopted, and an open source dataset labeling model with an open license agreement is deployed to each worker node in the distributed scheduling cluster. The open source dataset labeling model, such as a YOLO series model or a BERT series text classification model, is usually personalized trained or fine-tuned by a user in a dedicated cluster based on an open source framework (such as TensorFlow or PyTorch), and has stronger task specificity. According to the model type and version information specified by the user in the configuration file, the corresponding model package is obtained from the model repository (such as ModelHub or a local private image repository), and the model is deployed to each worker node through the K8S scheduling strategy. After deployment is completed, the model listening service is started, and the service interface of each model instance is uniformly registered through the Ray Serve module, to realize dynamic calling and result feedback of the external task platform. The deployment process ensures exclusive running, resource sharing and data isolation of the model in the dedicated cluster.

[0094] The embodiment adapts to the differentiated needs of resource scheduling and model use for different types of tasks by distinguishing the model deployment strategy according to the type of container cluster. In a public container cluster, a platform preset model is deployed, which can improve resource utilization efficiency and model reusability; in a dedicated container cluster, an open source labeling model is deployed, which ensures model customization and deployment flexibility.

[0095] In one embodiment, for the above step S122, the open source dataset labeling model is deployed to each worker node in the distributed scheduling cluster, which is further described as follows. Based on the above embodiment, the following is included:

[0096] S1221, obtaining a model address of the open source dataset labeling model;

[0097] S1222, load the open source dataset annotation model into each worker node in the distributed scheduling cluster according to the model address, to realize deployment of the open source dataset annotation model to each worker node in the distributed scheduling cluster.

[0098] In the embodiment, the model address of the open source dataset annotation model is obtained, the model address being identification information indicating the location of the model file in a storage system or a remote repository, containing an access protocol, path information and a version number. The model address can be filled in a model location field by a user through a WEB front end when creating an annotation task, or can be automatically generated after model uploading is completed. After the model address is read, the accessibility of the address and the file integrity are checked to confirm that the model file can be read and used for subsequent loading.

[0099] After the model address is confirmed, the open source dataset annotation model is loaded into each worker node in the distributed scheduling cluster according to the model address. Specifically, through an Actor task in the Ray cluster, a model loading process is triggered in each worker node. The model loading process includes pulling the model file from the model address, verifying the model integrity, and registering the model to a service interface. Specifically, the model file is downloaded to the local cache directory of each node through a distributed file transfer mechanism (such as HTTP, S3 protocol or K8S mounted volume), and is verified (for example, consistency is verified through SHA256 value) to ensure the consistency and reliability of the loaded model. Subsequently, the model structure and weight are parsed, and the model is loaded as a callable service through a compatible framework, and deployment initialization is completed.

[0100] After the model is loaded, the model service interface of each worker node is registered through Ray Serve, including service name, interface path, calling mode and other information. The distributed deployment strategy combines the task scheduling mechanism of Ray and the service discovery capability of K8S, so that the open source annotation models deployed on multiple worker nodes can respond to external requests in a load balancing manner, realizing task parallel processing and full utilization of computing resources.

[0101] In one embodiment, the above step S14 is split for storage, and an implementation is provided herein. Based on the above embodiment, it includes:

[0102] S141, upload the effective dataset to the object storage to perform splitting processing on the effective dataset, to obtain multiple groups of encrypted data after splitting;

[0103] S142, encrypt each encrypted data to obtain encrypted data, and store each encrypted data in the object storage in the form of an encrypted file, to obtain a dataset to be annotated stored in the object storage.

[0104] In this embodiment, the effective data set is uploaded to the object storage, specifically, the original data file after preliminary screening and preprocessing is uploaded to the preset object storage bucket through the standardized upload interface (such as S3 protocol, OSS interface, MinIO API, etc.). The object storage can use a distributed storage system to realize high availability and high concurrency access of data. Then, the uploaded effective data set is subjected to splitting processing in the object storage. The splitting operation is performed according to a preset splitting strategy, for example, grouping according to data volume (such as 1000 groups per group), data size (such as 100MB per group) or content structure. The split data unit is called encrypted data, and each group of encrypted data contains a plurality of original data samples and is organized in a unified format for subsequent encryption. For each group of encrypted data, encryption is performed. Encryption can use symmetric encryption algorithm (such as AES-256), asymmetric encryption algorithm (such as RSA) or hybrid encryption scheme. Each group of encrypted data forms a group of encrypted data after encryption, and is encapsulated as an encrypted file. The encrypted file contains encrypted data and encryption metadata (such as version number, data digest, encryption method identifier, etc.), which is convenient for subsequent verification and decryption.

[0105] Finally, each encrypted file is re-uploaded and stored back to the object storage to form a structured data set to be labeled. The object storage system provides traceable path information for each group of encrypted files according to the set naming rules or index information. By splitting and encrypting the effective data set and storing it in the object storage in the form of an encrypted file, the secure isolation and parallel splitting preprocessing before data processing are effectively realized, providing a unified, controllable and secure data input mechanism for subsequent large-scale labeling tasks, and enhancing the data protection capability and task scheduling flexibility.

[0106] In one embodiment, the data preprocessing of the original data set in step S13 is further described. Based on the above embodiment, it includes:

[0107] S131, the original data set is subjected to data deduplication processing to obtain a first intermediate data set.

[0108] Specifically, the original data set is subjected to repetitive identification and elimination. Repetition can be judged based on one or more of file content checksum, file name and time information, sample primary key and content similarity. For example, an index is constructed based on content hash (such as MD5, SHA256), text semantic similarity (such as cosine similarity) or structural features (such as timestamp sequence) to identify duplicate samples and eliminate redundant items, and output the first intermediate data set. By eliminating duplicate samples, the size of the set and the redundancy are reduced, and the efficiency of subsequent data processing is improved.

[0109] S132, performing data desensitization processing on the first intermediate data set to obtain a second intermediate data set.

[0110] Specifically, data desensitization refers to operations such as replacement, masking, or encryption on data fields containing sensitive information; it also includes deleting or masking fields related to identity, contact information, location information, or other fields that need to be desensitized; exemplary desensitization operations include replacing names with a uniform alias and replacing ID numbers with a hash value. At the same time, data strongly associated with sensitive information is filtered and isolated, and the output is a second intermediate data set. By removing sensitive identification content, user privacy is protected on the basis of data availability, and the compliance and shareability of the data set are improved.

[0111] S133, performing data detoxification processing on the second intermediate data set to obtain a third intermediate data set.

[0112] Specifically, the second intermediate data set is identified and data containing unfriendly comments or possible security issues are removed. Exemplary, according to the pre-set judgment rule, the content meeting the judgment rule is excluded or replaced; such as, through keyword blacklist matching to realize content screening, and obtain a third intermediate data set. By filtering possible bad data, the compliance and usability of the data content are improved.

[0113] S134, performing data enhancement processing on the third intermediate data set to obtain an effective data set.

[0114] Specifically, the enhancement method selects the appropriate combination of transformations according to the data type, for example, for text, symbol normalization and sample expansion can be performed, for images, size normalization and image quality adjustment can be performed, for audio, amplitude correction and noise reduction can be performed, and for video, resolution unification and key frame processing can be performed, and the output is an effective data set. By normalizing and moderately expanding the structure and performance of the data, consistent input and stability for subsequent processing are achieved.

[0115] Through the sequential data deduplication, desensitization, detoxification, and enhancement processing, an effective data set is finally formed. The effective data set removes redundancy and possible problem information, improves data credibility and applicability, and meets the requirements of data labeling and model training.

[0116] In one specific embodiment, the original data set includes text data, image data, audio data, and video data. On this basis, the data preprocessing of the original data set is further expanded. Figure 2 The method flowchart for data preprocessing of the original data set provided by the embodiments of the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the preprocessing further includes:

[0117] S21, remove non-text data or special characters from the text data to obtain valid text data;

[0118] S22, perform image quality adjustment processing on the image data to obtain valid image data;

[0119] S23, perform noise reduction processing on the audio data to obtain valid audio data;

[0120] S24, perform video quality adjustment processing on the video data to obtain valid video data.

[0121] In this embodiment, first, each type of data in the original data set is classified and identified to determine whether it belongs to text data, image data, audio data or video data, and the corresponding data preprocessing strategy is adapted. In the preprocessing process of text data, through regular expression matching, text segmentation and stop word filtering, etc., non-text elements (such as label symbols, HTML code segments, emoticons, etc.) and special characters (such as control characters, special symbols) are removed, so as to extract structured valid text data and improve the accuracy of subsequent semantic modeling. In the preprocessing of image data, according to the image resolution and contrast information, image sharpening, contrast enhancement, color balance and other image enhancement algorithms are used to uniformly adjust the image quality to eliminate blur, low exposure or distortion, and ensure the stability of the image feature extraction process. In the processing of audio data, spectrum analysis combined with spectral subtraction, adaptive filtering and other noise reduction algorithms are used to remove background noise, echo or electromagnetic interference, improve speech clarity, and ensure the extraction quality of audio features. In the preprocessing process of video data, by dividing the video frame sequence, sampling and uniformly processing the quality, video denoising, brightness / contrast adjustment and inter-frame enhancement strategies are applied to improve the overall clarity and stability of the video data, so as to ensure that discriminative feature sequences can be extracted in the subsequent processing link. Through the above-mentioned targeted preprocessing steps of multi-modal data respectively, high-quality preliminary cleaning and standardization processing of the original data set are realized, which lays a solid foundation for generating subsequent valid data sets.

[0122] Figure 3 The method for adjusting the labeled data provided by the embodiments of the present application is shown in the flowchart. After the decrypted data set is input into the target labeling model to generate labeled data, the labeled data can be further adjusted. Based on the above-mentioned embodiments, as shown in Figure 3 the adjustment of the labeled data includes:

[0123] S31, the labeled data is processed to obtain the labeling adjustment operation performed by the user in response to the labeled data, and the adjusted labeled data is obtained;

[0124] S32, the adjusted labeled data is stored in the object storage.

[0125] In the embodiment, the annotated data is presented graphically or interactively through a visual display interface (WEB front end) for users to review and confirm the current annotation results. The display process can adapt different display methods according to the data type: for text data, provide highlighted display and editable text area of annotation results; for image data, superimpose bounding box, mask or key point information to intuitively present the annotation area; for audio and video data, display the label position and content through the time axis method, and support frame-by-frame or segment-by-segment review combined with the play control component. During the display process, the user can also adjust the annotation results through the graphical interface, shortcut keys or semantic instruction methods, such as redrawing the image bounding box, modifying the text classification label, adjusting the time period annotation, etc.

[0126] When the user completes the inspection and adjustment of the annotated data, all annotation adjustment operations of the user are collected and recorded, and the original annotated data is updated to obtain the adjusted annotated data. To ensure data consistency and version management, the adjusted annotated data is associated with the original annotated data in version, and is encapsulated in a structured format (such as JSON or a custom label format) and written into an object storage service. During storage, metadata such as modification time, modification user, and adjustment content summary can also be recorded to support subsequent viewing and analysis. The annotation feedback mechanism with user participation in the embodiment further improves the accuracy of data annotation and the efficiency of constructing high-quality data sets.

[0127] Further, it also includes a WEB front end for task fine-grained configuration. That is, the user can set or select the requirements in the front-end interface. Further, the annotation task information is formed.

[0128] In the embodiment, the WEB front end is used as a human-computer interaction interface component, and the user sets and selects various configuration parameters of the annotation task in the visual operation interface. The WEB front end integrates a task creation module, a configuration item management module and a real-time preview module. The user can input or select the task name and the identification information of the annotated data set through the task creation module; through the configuration item management module, the selection items including but not limited to data type (such as text, image, audio or video), annotation method (such as classification, named entity recognition, bounding box annotation, key point annotation, etc.), annotation range (such as all or a specified subset in the data set), selection of annotation model (such as whether to combine a pre-set model for intelligent auxiliary annotation) and initialization setting of the data set label system are set item by item.

[0129] Then, for text type data, the WEB front end further provides a regular field configuration interface, and the user can set text preprocessing rules such as sensitive field identification and formatting cleaning rules by inputting a regular expression, so as to improve the consistency and accuracy of subsequent labeling process. The front end can also configure a preset template and import an external configuration file to facilitate quick reuse in a multi-task configuration scenario. After completing the fine-grained configuration, the WEB front end structures and encapsulates all configuration contents into labeling task information, and submits it to the task management module to drive the subsequent labeling task scheduling and execution process. Through the user-oriented graphical interaction and configuration driving mechanism, the WEB front end realizes flexible definition and customization of labeling tasks, effectively improving the ease of use, configuration efficiency and ability to adapt to diversified scenarios.

[0130] In another specific embodiment, in the labeling execution phase, in addition to using the preset labeling model to automatically label the data to be labeled, a manual labeling method is also provided, which includes individual labeling and team labeling. Individual labeling refers to that a single labeling personnel performs labeling operation on the data to be labeled based on the front-end interaction interface or local tools, assigns an independent labeling task unit to the data to be labeled, and records the operation track and labeling result. Team labeling refers to that multiple labeling personnel participate in the same labeling task, and according to a preset task allocation strategy, the data to be labeled is processed in fragments, and is dynamically allocated to different members according to the member labeling ability or task progress.

[0131] Further, in the team labeling process, the team collaboration mechanism is further supported, including labeling progress visualization display, labeling consistency checking mechanism, multi-person labeling conflict detection and result fusion strategy. Labeling review personnel can be set to randomly check or fully review the labeling results submitted by team members, so as to realize multiple rounds of quality control. At the same time, the team labeling supports user permission hierarchical configuration, which distinguishes labeling members, review members and management members, so as to facilitate task scheduling and quality responsibility division. The manual labeling mechanism and the model labeling mechanism are independent and can be combined for use, and can be flexibly configured according to data complexity, scene demand and model availability, and can also be used in various combination modes such as manual assistance to machine labeling, machine assistance to manual labeling and manual review, so as to ensure labeling efficiency while considering data quality and labeling consistency.

[0132] Figure 4 The structural schematic diagram of the data set labeling device provided in the present application is shown in Figure 4 As shown in the figure, the data set labeling device 40 provided in the embodiment includes:

[0133] The information acquisition module 401 is configured to acquire the model modification parameter of the preset artificial intelligence model modified by the user and / or the open source data set labeling model uploaded by the user.

[0134] The deployment execution module 402 is configured to deploy the open source data set annotation model to the preset container cluster and / or deploy the preset artificial intelligence model to the preset container cluster according to the model modification parameter;

[0135] The information processing module 403 is configured to obtain an original data set, perform data preprocessing on the original data set, and obtain an effective data set;

[0136] The information processing module 403 is further configured to upload the effective data set to an object storage for split encryption storage, and obtain a to-be-annotated data set stored in the object storage;

[0137] The information processing module 403 is further configured to obtain annotation task information and determine an annotation mode according to the annotation task information;

[0138] The annotation execution module 404 is configured to, when it is detected that the annotation mode is model annotation, call the to-be-annotated data set from the object storage according to the annotation task information, and call a preset artificial intelligence model and / or an open source data annotation model as a target annotation model from the preset container cluster according to the annotation task information;

[0139] The annotation execution module 404 is further configured to decrypt the to-be-annotated data set, and obtain a decrypted data set;

[0140] The annotation execution module 404 is further configured to input the decrypted data set into the target annotation model, generate annotated data, and store the annotated data in the object storage.

[0141] In an embodiment, the preset container cluster is a public container cluster or a private container cluster, and the preset container cluster is deployed with a distributed scheduling cluster, and the distributed scheduling cluster includes one control node and at least one worker node.

[0142] The deployment execution module 402 is specifically configured to:

[0143] When it is detected that the preset container cluster is a public container cluster, the preset artificial intelligence model is deployed to each worker node in the distributed scheduling cluster according to the model modification parameter, so as to realize distributed deployment of the preset artificial intelligence model to the public container cluster; when it is detected that the preset container cluster is a private container cluster, the open source data set annotation model is deployed to each worker node in the distributed scheduling cluster, so as to realize distributed deployment of the open source data set annotation model to the private container cluster.

[0144] Further, the deployment execution module 402 is specifically configured to:

[0145] Obtain a model address of the open source dataset labeling model; load the open source dataset labeling model into each work node in the distributed scheduling cluster according to the model address, so as to deploy the open source dataset labeling model to each work node in the distributed scheduling cluster.

[0146] In one embodiment, the information processing module 403 is specifically configured to:

[0147] Upload the valid dataset to the object storage for splitting processing of the valid dataset to obtain a plurality of groups of encrypted data after splitting; encrypt each encrypted data to obtain encrypted data, and store each encrypted data in the object storage in the form of an encrypted file to obtain the encrypted dataset stored in the object storage.

[0148] In one embodiment, the information processing module 403 is further specifically configured to:

[0149] Perform data deduplication processing on the original dataset to obtain a first intermediate dataset; perform data desensitization processing on the first intermediate dataset to obtain a second intermediate dataset; perform data detoxification processing on the second intermediate dataset to obtain a third intermediate dataset; and perform data enhancement processing on the third intermediate dataset to obtain the valid dataset.

[0150] In one specific embodiment, the original dataset includes text data, image data, audio data and video data.

[0151] The information processing module 403 is specifically configured to:

[0152] Remove non-text data or specific characters from the text data to obtain valid text data; perform image quality adjustment processing on the image data to obtain valid image data; perform noise reduction processing on the audio data to obtain valid audio data; and perform video quality adjustment processing on the video data to obtain valid video data.

[0153] In one embodiment, the method further comprises an exhibition processing module; the exhibition processing module is configured to perform exhibition processing on the labeled data to obtain a labeling adjustment operation performed by a user in response to the labeled data, and obtain adjusted labeled data; and store the adjusted labeled data in the object storage.

[0154] The dataset labeling apparatus 40 provided in this embodiment can execute the method provided in the method embodiment, and has similar implementation principles and technical effects, which will not be described here in detail.

[0155] Figure 5 The structural schematic diagram of the electronic device provided in the present application is shown in FIG. 1. Figure 5As shown, the electronic device 50 provided by the embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected through a bus 504.

[0156] In the implementation process, the at least one processor 501 executes the computer execution instructions stored in the memory 502, so that the at least one processor 501 executes the above-mentioned method.

[0157] The specific implementation process of the processor 501 can refer to the above-mentioned method embodiment, which has similar implementation principles and technical effects, and will not be described here in detail.

[0158] In the above-mentioned embodiments, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, for short: CPU), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, for short: DSP), application specific integrated circuits (English: Application Specific Integrated Circuit, for short: ASIC) and the like. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0159] The memory can contain a random access memory (Random Access Memory, RAM), and can also include a non-volatile memory (Non-volatile Memory, NVM), for example, at least one disk memory.

[0160] The bus can be an industry standard architecture (Industry Standard Architecture, ISA) bus, a peripheral component interconnect (Peripheral Component, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit only one bus or one type of bus.

[0161] The present application also provides a computer program product, including a computer program, which is executed by a processor to implement the above-mentioned method.

[0162] The application further provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions.

[0163] The readable storage medium can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0164] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0165] The division of units is only a logical function division, and in actual implementation, there can be another division mode, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0166] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0167] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0168] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0169] It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0170] Finally, it should be noted that: those skilled in the art will easily think of other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application that follow the general principles of the present application and include common knowledge or conventional technical means in the art that are not disclosed in the present application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

Claims

1. A dataset annotation method, characterized in that, include: Obtain the model modification parameters of the user-modified pre-built artificial intelligence model and / or the user-uploaded open-source dataset labeled model; Deploy the open-source dataset annotation model to a preset container cluster and / or modify the parameters according to the model to deploy the preset artificial intelligence model to a preset container cluster; Obtain the original dataset, perform data preprocessing on the original dataset, and obtain a valid dataset; The valid dataset is uploaded to object storage for splitting and encryption storage, resulting in the unlabeled dataset stored in the object storage. Obtain annotation task information, and determine the annotation method based on the annotation task information; When the annotation method is detected to be model annotation, the dataset to be annotated is retrieved from the object storage according to the annotation task information, and the preset artificial intelligence model and / or open source data annotation model is retrieved from the preset container cluster as the target annotation model according to the annotation task information. The dataset to be labeled is decrypted to obtain the decrypted dataset; The decrypted dataset is input into the target annotation model to generate labeled data, and the labeled data is stored in the object storage.

2. The method according to claim 1, characterized in that, The preset container cluster is either a public container cluster or a dedicated container cluster. A distributed scheduling cluster is deployed in the preset container cluster. The distributed scheduling cluster includes a control node and at least one worker node. The step of deploying the open-source dataset annotation model to a preset container cluster and / or deploying the preset artificial intelligence model to a preset container cluster by modifying the parameters according to the model includes: When the preset container cluster is detected to be a public container cluster, the preset artificial intelligence model is deployed to each working node in the distributed scheduling cluster according to the model modification parameters, thereby realizing the distributed deployment of the preset artificial intelligence model to the public container cluster. And / or when the preset container cluster is detected to be a dedicated container cluster, the open-source dataset annotation model is deployed to each worker node in the distributed scheduling cluster, thereby realizing the distributed deployment of the open-source dataset annotation model to the dedicated container cluster.

3. The method according to claim 2, characterized in that, The step of deploying the open-source dataset annotation model to each worker node of the distributed scheduling cluster further includes: Obtain the model address of the labeled model in the open-source dataset; The open-source dataset annotation model is loaded into each worker node of the distributed scheduling cluster according to the model address, so as to deploy the open-source dataset annotation model to each worker node of the distributed scheduling cluster.

4. The method according to claim 1, characterized in that, The step of uploading the valid dataset to object storage for splitting, encryption, and storage, to obtain the unlabeled dataset stored in the object storage, includes: The valid dataset is uploaded to object storage to split the valid dataset into multiple sets of data to be encrypted. Each piece of data to be encrypted is encrypted to obtain encrypted data, and each piece of encrypted data is stored in object storage as an encrypted file to obtain the dataset to be labeled stored in the object storage.

5. The method according to claim 1, characterized in that, The step of preprocessing the original dataset to obtain a valid dataset includes: The original dataset is deduplicated to obtain a first intermediate dataset; The first intermediate dataset is subjected to data anonymization processing to obtain the second intermediate dataset; The second intermediate dataset is depollinated to obtain the third intermediate dataset. The third intermediate dataset is augmented to obtain a valid dataset.

6. The method according to claim 5, characterized in that, The original dataset includes text data, image data, audio data, and video data; The data preprocessing of the original dataset further includes: Remove non-text data or specific characters from the text data to obtain valid text data; The image data is subjected to image quality adjustment processing to obtain effective image data; The audio data is subjected to noise reduction processing to obtain valid audio data; The video data is processed to adjust its quality, resulting in valid video data.

7. The method according to any one of claims 1 to 6, characterized in that, After inputting the decrypted dataset into the target annotation model to generate labeled data, the process further includes: The labeled data is displayed and processed to obtain the user's response to the labeled data and the label adjustment operation performed on the labeled data, thus obtaining the adjusted labeled data; The adjusted annotation data is stored in the object storage.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images