Data service method and system with integrated "collection, labeling, generation, management and usage"

Abstract

Disclosed in the present invention are a data service method and system with integrated "collection, labeling, generation, management and usage". The method comprises: determining, by means of a user behavior, original data that needs to be captured, and obtaining an original data set in view of a data format determined by means of a labeling task; performing data generation on the basis of the original data set, so as to obtain a generated data set; in view of a labeling task configured by a user, using a trained data labeling model to label data in the original data set and data in the generated data set, so as to obtain a labeled data set; and after the data labeling model, the original data set, the generated data set and the labeled data set are stored in a data center, configuring a permission and a retrieval tag for the data labeling model or data, wherein the data center is provided with a data labeling interface for providing a model labeling service, and a model management interface for providing a model management service. The present invention can efficiently implement a complete workflow from data collection to data usage while providing a comprehensive service for data.

Description

An integrated data service method and system for "standardization, management and use" Technical Field

[0001] The present invention relates to the technical field of data services, and in particular to a data service method and system. Background Art

[0002] The internet era is developing rapidly, and the amount of information online is exploding under the backdrop of big data. Simultaneously, deep learning technology is gaining widespread application and continued development across industry and academia. The application of deep learning in various industries requires large amounts of high-quality, labeled data. However, data is always limited in different scenarios. The process from acquiring data to generating labeled data to training deep learning models often involves multiple steps, which are fraught with difficulties and can consume significant time. Therefore, how to build a comprehensive data service system around massive amounts of data, covering the entire data lifecycle and providing comprehensive services for multiple data types to maximize data value, has long been a challenge for the industry.

[0003] Currently, the systems that provide services for massive data mainly include data annotation systems (such as Chinese-Annotator, Label-studio, and Doccano) and data management systems ("A Comprehensive Big Data Management System" (CN202210564880)). In addition, the main tools for data collection include Octopus, Content Grabber, Parsehub, and Mozenda. These systems or tools are usually only responsible for part of the data life cycle. A single function cannot provide comprehensive services for the data, and often requires manual connection to enable the joint use of multiple systems or tools.

[0004] In summary, current mainstream data service systems and tools offer only limited, unsustainable functionality and fail to cover the entire data lifecycle, significantly reducing the value of data. In order to combine data services provided by different platforms, users often need to perform extensive integration work, spending significant time on data processing and integration, resulting in low efficiency.

[0005] Summary of the Invention

[0006] In response to the above problems, the present invention provides an integrated data service method and system for "standard collection, management and use". The system uses data collection tools, data association subsystems, data generation services, data annotation subsystems and model deployment and scheduling services to provide comprehensive services for data, and can efficiently complete the entire work from data collection to data use.

[0007] The technical contents of the present invention include:

[0008] An integrated data service method for "standardization, management and application" includes:

[0009] Determine the raw data that needs to be captured through user behavior, and combine it with the data format determined by the labeling task to obtain the original data set;

[0010] Generating data based on the original data set to obtain a generated data set; wherein the data format in the generated data set satisfies the data format determined by the labeling task;

[0011] In combination with the labeling task configured by the user, the trained data labeling model labels the data in the original data set and the generated data set to obtain a labeled data set;

[0012] After the data annotation model, original data set, generated data set and annotated data set are stored in the data center, the permissions and retrieval tags of the data annotation model or data are set; wherein, the data center is provided with a data annotation interface that provides model annotation services and a model management interface that provides model management services.

[0013] Furthermore, the raw data to be captured is determined by user behavior, and combined with the data format determined by the labeling task to obtain the raw data set, including:

[0014] Obtaining a labeling task created by a system administrator, wherein the labeling task includes a data format and a labeling user corresponding to the labeling task, wherein the data format includes: a data type and a number of data items;

[0015] When a labeling user logs in, the user identity corresponding to the labeling user is verified and all labeling tasks corresponding to the labeling user are obtained;

[0016] Based on the annotation task selected by the annotation user, the data type and number of data items collected are limited;

[0017] Monitor user behavior and capture web page areas and content by limiting the type and number of data items collected to obtain the original data set.

[0018] Furthermore, the monitoring of user behavior and capturing of web page areas and content in combination with limiting the types and number of data items collected to obtain the original data set include:

[0019] When the data type is text data and the number of data items is a single item, monitoring whether the corresponding shortcut key is pressed after the user uses the mouse to select a section of text content, and capturing the text content selected by the mouse when the corresponding shortcut key is pressed;

[0020] When the data type is image data and the number of data items is a single item, monitoring whether a preset shortcut key is pressed, and when the preset shortcut key is pressed, capturing a picture of the image area where the cursor is located or starting a screenshot function;

[0021] When the number of data items is in batches, the HTML source code of the current page is obtained, and when the corresponding shortcut key is pressed, the xpath of the element in the cursor area in the web page DOM is analyzed, and the web page is re-rendered to highlight the element node content with the same xpath.

[0022] Furthermore, the data generation process includes:

[0023] Based on the data falsification tool, data enhancement is performed on the data in the original data set;

[0024] and,

[0025] Based on a data adversarial generation tool, adversarial samples of the data in the original data set are generated.

[0026] Furthermore, the trained data annotation model annotates the data in the original dataset and the generated dataset in combination with the user-configured annotation task to obtain an annotated dataset, including:

[0027] Combined with the user-configured annotation task, a large model is called to implement data pre-annotation; wherein, the large model includes: the online large model ChatGPT or the locally deployed large model ChatGLM;

[0028] After the annotation user verifies the pre-annotation results, the data annotation model is trained based on the verified results to obtain a trained data annotation model;

[0029] The data in the original dataset and the generated dataset are labeled based on the trained data labeling model to obtain a labeled dataset.

[0030] Furthermore, the method combines the user-configured annotation task and calls the large model to implement data pre-annotation, including:

[0031] Obtaining annotation configuration information input by the user; wherein the annotation configuration information is used to clarify the annotated data and task type, as well as various attributes involved in the annotation process;

[0032] Based on the annotation configuration information, fill the data to be annotated into a predefined prompt template to generate a prompt corresponding to the task;

[0033] Translate the prompt into several languages, extract a piece of data from the user's manually annotated data as a sample, and translate it into the corresponding language;

[0034] After embedding the sample of each language into the prompt, it is input into the large model as guidance information, so that the large model receiving the guidance information can annotate the data to be annotated based on the prompt of the corresponding language, and obtain the annotation result of the corresponding language after processing the redundant data in the result returned by the large model;

[0035] Translate the annotation results in the corresponding language into the language of the data to be annotated, and obtain a number of answers in the same language as the data to be annotated;

[0036] The multiple answers with the same language as the data to be annotated and the prompt with the same language as the data to be annotated are input into the large model, so that the large model selects one answer from the multiple answers with the same language as the data to be annotated as the annotation result of the data to be annotated.

[0037] Furthermore, after the large model selects one answer from the plurality of answers in the same language as the data to be annotated as the annotated result of the data to be annotated, the large model further includes:

[0038] Display the annotation results of the data to be annotated to the user so that the user can make revisions;

[0039] Add the revised annotation results to the next prompt to annotate the next data to be annotated.

[0040] Furthermore, the process of providing model management services based on the model management interface includes:

[0041] Based on the management request received by the model management interface, the packaged tarball is passed to the middleware; wherein the packaged tarball is compressed based on the model file, model parameters, control file, dependency files, and configuration file specified in the management request. The model file refers to a model structure file defined using Pytorch, the model parameters are serialized files of the model, the control file defines a class containing Python methods with different functions, and the configuration file is generated based on the model file name, model parameter name, and model version input by the user;

[0042] A process is started in the backend component to decompress the packaged tarball and extract all files into a temporary folder;

[0043] The backend component reads the configuration file and loads relevant files according to the model file name, model parameter name and control file name recorded in the configuration file to generate a model instance;

[0044] The backend component starts a thread in the frontend component according to the model file name recorded in the configuration file to register the model; each thread in the frontend component corresponds to a model instance process in the backend component;

[0045] When the front-end component receives a model inference request, it parses the model inference request to obtain input data;

[0046] Processing the input data according to the preprocessing function specified in the control file to obtain preprocessed data;

[0047] Input the preprocessed data into the model for inference, and perform data post-processing on the inference results;

[0048] The data is processed and returned to the user.

[0049] Furthermore, the method also includes: training the model based on passive full-scale training and / or active learning iterative training.

[0050] The integrated data service system for "standard collection and management" of the present invention includes:

[0051] Data collection tools are used to determine the raw data that needs to be captured based on user behavior and combine it with the data format determined by the annotation task to obtain the original data set;

[0052] A data generation service, configured to generate data based on the original data set to obtain a generated data set; wherein the data format in the generated data set satisfies the data format determined by the annotation task;

[0053] A data annotation subsystem, configured to annotate the data in the original dataset and the generated dataset using a trained data annotation model in combination with a user-configured annotation task to obtain an annotated dataset;

[0054] The data management subsystem is used to store the data annotation model, original data set, generated data set and annotated data set in the data center, and then set the permissions and retrieval tags of the data annotation model or data; wherein, the data center is provided with a data annotation interface that provides model annotation services and a model management interface that provides model management services.

[0055] Compared with the existing technology, the method provided by the present invention has the following advantages in providing data services:

[0056] 1. The Internet collection plug-in collects specified data from the Internet based on user behavior. It can accurately collect both bulk data and sparse data on structured web pages, quickly and efficiently.

[0057] 2. The browser collection plug-in and the data annotation system are linked together, and the collected content is automatically converted into the data format required by the annotation system. After the user pushes the data to the annotation system, he can start annotation directly in the system without the need for data preprocessing, which reduces the user's time cost.

[0058] 3. The browser collection plug-in is oriented to text and image data, realizing single text, single image, and text-image data collection methods, and can be used for text annotation, image annotation, and image-text cross-modal annotation.

[0059] 4. The model deployment and scheduling service provides automated model deployment and scheduling capabilities, enabling linkage with the data management system. Users can complete model training using data within the data management system and deploy the trained model with one click, reducing the complexity of user model development and improving user work efficiency.

[0060] 5. The model deployment and scheduling service is linked to the annotation system, enabling a collaborative human-machine annotation model. This model improves its capabilities by actively learning from user annotation results and provides pre-annotation support, improving user annotation efficiency. After annotation is complete, both the annotated data and the model trained using that data are obtained. The pre-annotated model can be used to provide services supporting related businesses.

[0061] 6. Data generation service provides comprehensive data generation services for image, video, audio, and text data. Users can use the service to generate large amounts of data to expand data and solve the problem of data scarcity.

[0062] 7. The annotation system provides a variety of annotation tools and task templates for all types of data, which can widely support work in various annotation scenarios.

[0063] 8. The data service system for standardization and management covers the entire process from data acquisition to data labeling, management, data use and generation, maximizing the value of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] FIG1 is a data flow diagram of the data service system of the present invention.

[0065] Figure 2. Architecture diagram of the data collection tool.

[0066] Figure 3 Data flow diagram of data acquisition.

[0067] Figure 4: Architecture diagram of the data annotation subsystem.

[0068] Figure 5 Data flow diagram for data annotation.

[0069] Figure 6 Flowchart of the cross-language capability and self-consistency processing of the large model.

[0070] Figure 7 Auxiliary annotation flow chart based on large model.

[0071] Figure 8: Architecture diagram of model deployment and scheduling services.

[0072] Figure 9 Data flow diagram of active learning iterative training.

[0073] Figure 10: TorchServe architecture diagram.

[0074] Figure 11 Data flow diagram of the data service. DETAILED DESCRIPTION

[0075] In order to make the above features and advantages of the present invention more obvious and easy to understand, the technical solutions of the present invention are further described below through specific embodiments.

[0076] The integrated data service system for "collection, standardization, production, management and use" of the present invention includes data acquisition tools, data annotation subsystem, data generation service, data management subsystem, model deployment and scheduling services to cover the entire life cycle of data "collection, standardization, production, management and use". From the perspective of data flow, the system's data flow diagram is shown in Figure 1.

[0077] 1. Data collection tools.

[0078] The data collection tool is the "collection" part of the data lifecycle. It can be used to collect data that is widely available on structured web pages in batches, or to collect small amounts of data from the internet individually. The data collection tool communicates with the data annotation subsystem, formatting the collected data and pushing it to the data annotation system, eliminating the need for cleaning and preprocessing the collected data.

[0079] In one embodiment, the data collection tool may be a browser (Chrome) plug-in, as shown in FIG2 . The architecture of the browser plug-in includes a user login module, a task selection module, a behavior detection module, and a data transmission module.

[0080] ●User login module.

[0081] The user login module is used to verify the user's identity in the annotation system and the annotation tasks the user is participating in. In this embodiment, the user login module uses the same authentication system as the data annotation system, shares the same database and browser login information, and the user only needs to log in to either the annotation system or the plug-in to complete single sign-on.

[0082] ●Task selection module.

[0083] The task selection module is used by the plug-in to select the annotation task for data collection. Since different tasks require different data formats and the amount of data required is also different, for example, text classification requires one text, text generation requires two texts, and image classification requires one image, etc. Therefore, this module fixes the input format of the data by selecting different annotation tasks, so that the collected data can be directly applied to data annotation.

[0084] ●Behavior monitoring module.

[0085] The behavior monitoring module is used to monitor user behavior. Users can wake up the plug-in through shortcut keys, etc. The plug-in determines the web page area and content that needs to be captured based on user behavior.

[0086] Among them, the behavior monitoring module provides two data collection methods: single collection and batch collection.

[0087] In the single-line collection method: for text data, after the user uses the mouse to select a piece of text content, the module monitors whether the preset shortcut key is pressed. When the user wakes up the plug-in through the shortcut key, the module automatically captures the content selected by the mouse; for image data, two collection methods are provided: the first is when the user moves the mouse to the image area and presses the preset shortcut key to wake up the plug-in, the plug-in captures the original address of the image and grabs the image; the second is when the user presses the preset shortcut key to wake up the plug-in screenshot function, and the user manually captures the image area.

[0088] In batch collection mode, the module automatically obtains the HTML source code of the current page. The user wakes up the plug-in through the shortcut key and clicks on the web page content. The module automatically analyzes the xpath of the element in the cursor area in the web page DOM and re-renders the web page. The element node content with the same xpath is highlighted to facilitate the display of batch collection results to the user. Collection results with different xpath elements require different fields to be defined.

[0089] ●Data transmission module.

[0090] The data transmission module is used to transmit the collected data in a fixed data format to the data storage module of the data annotation system. Since the task selection module has determined the input format of the data in advance according to the task type, the transmitted data meets the format requirements of a single task data, and the data can be directly used in the data annotation process.

[0091] This embodiment uses a data collection tool to complete the data collection process, as shown in Figure 3, including:

[0092] 1) The system administrator creates and configures annotation tasks in the annotation system, and specifies the data format (data type and number of data items) of the annotation tasks. This configuration determines the subsequent format requirements for the collected data.

[0093] 2) The labeling user logs in to the browser plug-in, which verifies the user's identity and obtains all the user's labeling tasks.

[0094] 3) The user selects the annotation task. Different annotation tasks have different requirements for data formats. The plug-in limits the type and number of data collected by the user based on the requirements of the data format. Data that does not meet the requirements cannot be inserted into the annotation system.

[0095] 4) Mark users to wake up the plug-in in different ways to collect text or image data.

[0096] 5) When the collected data meets the format requirements of a single piece of data for the annotation task, the user can insert the data into the annotation system.

[0097] 2. Data generation services.

[0098] The data generation service is the "creation" part of the data lifecycle, providing multiple types of data generation interfaces for using the data in the data management subsystem to generate additional data in different scenarios to expand the original data set.

[0099] In one embodiment, the data generation service of the present invention can be used as a standalone platform to provide anti-attack and anti-defense services, or it can be embedded in a data management system, using the data set within the data management system to call the service to generate additional data to achieve the purpose of expanding data. The service includes a tool management module, a tool status monitoring module, an authorization authentication module, and a data generation module:

[0100] ●Tool management module.

[0101] The tool management module is used to add and delete data generation tools, as well as adjust the online and offline status of each tool.

[0102] ●Tool status monitoring module.

[0103] The tool status monitoring module is used to monitor the operating status of data generation tools and check the usage of each tool.

[0104] ●Authorization authentication module.

[0105] The permission authentication module is used to verify user permissions. This module uses the same authentication system as the data management subsystem and data labeling subsystem to share user information.

[0106] ●Data generation module.

[0107] The data generation module integrates all data generation tools such as images, text, video, audio, etc., and is used to receive data and generate additional data.

[0108] The data generation module considers two usage scenarios. The first is when users lack data. This service integrates a data forgery tool, allowing users to augment existing data and expand their datasets. The second is when methods constructed using existing data are ineffective and lack robustness in specific applications. This service integrates a data adversarial generation tool, allowing users to generate adversarial examples using existing data, increasing the number of adversarial examples in the dataset and, by incorporating them into the method construction process, improving the robustness of the method.

[0109] 3. Data annotation subsystem.

[0110] The data labeling subsystem is the "labeling" part of the data life cycle, which is used to complete data labeling work. It uses user uploads and imports from data management as data sources, supports text, image, video and audio file formats, and then goes through the labeling process of multiple teams, multiple users and multiple roles. Finally, the completed labeled data can be downloaded locally or stored in the data management subsystem.

[0111] In one embodiment, the architecture diagram of the annotation subsystem, as shown in FIG4 , includes a data annotation module, a data acceptance module, a team management module, a task management module, a user login module, and a data storage module.

[0112] ●Data labeling module.

[0113] The data annotation module is used to complete annotation tasks. When a task publisher assigns a task to a team or individual, the user completes the data annotation work in this module. Figure 5 shows the data flow diagram for data annotation. This module completes two review processes: audit and quality inspection. Both full and random audits are supported. Full audit refers to the entire data set, while random audit refers to a sample of data.

[0114] In the labeling module, when the user is performing the labeling process, the module will initially transmit the user's labeling results to the model deployment and scheduling service, waiting for the background model to complete active learning. Later, before the user loads the next piece of data, the module will transmit the unlabeled data to the background model, obtain the model's pre-labeling results and display them to the user. After the user completes the correction, the labeling results will be sent back to the background model. In this way, the accuracy of the background model is continuously improved, and the user's labeling efficiency will also be improved, achieving a human-computer collaborative labeling mode.

[0115] In addition to coupling with the model deployment and scheduling service to implement intelligent labeling, the labeling module can also pre-label data by invoking large models. Large models are deep learning models with enormous parameter sizes and computing power, resulting in more powerful representational and learning capabilities. This makes large models excellent for handling complex tasks, large datasets, and a wide range of domains. In this example, the online large model ChatGPT and the locally deployed large model ChatGLM are used as language models for auxiliary labeling.

[0116] In the large-model assisted annotation mode, first, you need to define the annotation task, clarify the annotated data and task type, as well as the various attributes involved in the annotation process. For example, for text extraction tasks, users need to configure the categories of entities and relationships. Then, based on the information configured by the user, the predefined prompt template in the system is filled in, and the prompt corresponding to the task is automatically generated. After the user manually annotates several pieces of data, the system will extract a piece of data as a sample and embed it into the prompt, which will be input into the large model as guiding information to obtain more accurate annotation results. For example, for an ongoing relationship extraction task, the prompt may look like this:

[0117] "I'd like you to be a data annotation engineer. I'll input some text data to be annotated, and your job is to annotate the head and tail entities and their relationships based on the input text. This work may include named entity recognition and classification of the relationships between the head and tail entities. The annotation results will be in the following format:

[0118] [{

[0119] "head":"...",

[0120] "tail":"...",

[0121] "relation":"..."

[0122] }]

[0123] Here is my text content:

[0124] American George F. Kennedy performed community service on August 22, 2023, at the Texas Book Depot in Dallas, Texas.

[0125] By processing the text returned by the large model and removing invalid text, the annotation results can be obtained as follows:

[0126] [{

[0127] "head":"George Kennedy",

[0128] "tail":"Texas Book Warehouse",

[0129] "relation":"community service"

[0130] }]

[0131] To improve the accuracy of large-scale model annotation, this method comprehensively utilizes the cross-language capability and self-consistency of the large-scale model to enhance its natural language understanding capability, thereby making the annotation results more accurate. As shown in Figure 6, the specific method is as follows:

[0132] First, since the training corpus of existing large models (such as ChatGPT) is usually composed of multiple languages, the more widely used languages ​​​​have more corpus, and the large model will have a stronger understanding of the more widely used languages ​​​​, so we use the world's most commonly used languages ​​​​as alternative languages ​​​​, namely Chinese, English, Spanish, French, and German. This method will translate the prompt input to the large model into the above five languages ​​​​and input them into the large model five times. The large model will produce different ways of thinking based on its understanding of different types of languages ​​​​and thus produce different answers.

[0133] Then, the results in different languages ​​are translated back into the language of the text to be annotated, and five answers in the same language as the text to be annotated are obtained. The five answers and the original prompt are then combined into a multiple-choice question, which is input into the big model for it to make a choice. This is equivalent to letting the big model act as an auditor of the five annotated answers, thereby selecting the better answer as the final result. For example:

[0134] "I want you to be a data annotation reviewer. I will input a piece of text data to be annotated and five entity relationship annotation results. Your job is to select the most accurate one from the five annotation results as the final result.

[0135] The text to be annotated is as follows:

[0136] American George F. Kennedy performs community service on August 22, 2023, at the Texas Book Depot in Dallas, Texas.

[0137] The five annotation results are as follows:

[0138] (a)…

[0139] (b)…

[0140] (c)…

[0141] (d)…

[0142] (e)…

[0143] Please give me the option to mark the results directly"

[0144] Finally, the system presents the final result selected by the large model to the user. After the user makes revisions, this information is added to the next prompt as guidance. This way, by inserting examples into the prompt, the large model returns more accurate results.

[0145] The auxiliary annotation process based on the large model, as shown in Figure 7, includes:

[0146] 1) User configures the annotation task;

[0147] 2) The system obtains the annotation configuration;

[0148] 3) The system obtains the current data to be labeled;

[0149] 4) Determine whether the current data to be labeled is labeled data;

[0150] 5) If yes, obtain the labeled data;

[0151] 6) If not, construct a Prompt, input the result of the Prompt into the large model, process the redundant data in the returned result, and obtain the formatted result.

[0152] ●Data acceptance module.

[0153] The data acceptance module is used to complete the acceptance of labeled data. After the data has been labeled and the related quality inspection work is completed in the annotation module, it is handed over to the acceptance personnel in the data acceptance module for final acceptance. The review process in quality inspection also includes full review and random review. At any review process, data that does not meet the requirements can be rejected and sent back to the annotator for re-labeling. The acceptance personnel should belong to the first party of the labeling task, and the labeling personnel should belong to the second party of the labeling task. If the labeling task does not have a designated acceptance personnel, the acceptance process is not required. The labeling task ends after the data labeling and related review work are completed.

[0154] ●Team management module.

[0155] The team management module is used by project administrators to create and manage teams. Project administrators can create accounts for all members in the annotation process and combine some members into a team. When assigning annotation tasks, they can select users or teams to participate in the annotation tasks.

[0156] In the team management module, the system divides user roles into project managers, labelers, reviewers, quality inspectors, and acceptance inspectors, corresponding to the creation, labeling, review, quality inspection, and acceptance in the labeling process. Users can choose different processes according to different task volumes. For example, large-scale labeling tasks can cover all processes, and small-scale labeling tasks can only include labeling and review.

[0157] ●Task management module.

[0158] The task management module is used by project administrators to create, modify, and delete annotation tasks. This system supports annotation of text, images, videos, and audio data. It supports users to upload data to be annotated and import data to be annotated from the data management system. It also provides task templates for various types of tasks, allowing users to freely select annotation scenarios and configure annotation tasks in detail.

[0159] In the task management module, this system provides a variety of annotation tools and task templates for different types of data. For example, it provides image data with task templates such as general image annotation and target detection, as well as annotation tools such as 2D boxes, 3D boxes, and point clouds. It also provides text with task templates such as named entity recognition and entity relationship annotation, as well as annotation tools such as fragment annotation, relationship annotation, and global attributes.

[0160] ●User login module.

[0161] The user login module is used to verify user identity and display different pages for users with different roles. This module and the data collection tool use the same authentication system, share the same database and browser login information.

[0162] ●Data storage module.

[0163] The data storage module is used to temporarily store the results of data annotation and regularly store the data persistently in the local system.

[0164] In the data storage module, considering that the system needs to respond to the annotation operation in a timely manner and has high efficiency requirements, this module uses Redis to temporarily store the annotation data and stores the data in the cache. Due to its fast read and write speed, it can meet user needs and persist the annotation data in the cache to the local system after the data reaches a certain level.

[0165] 4. Data management subsystem.

[0166] The data management subsystem is the "management" component of the data lifecycle. Serving as the system's data hub, it stores data from various data sources, including data collected by users through collection tools, data uploaded by users, data annotated by the annotation system, and data generated by the data generation service. The underlying system is an object storage service that supports the storage of any type of data and supports assigning permissions to data, setting visible users, and tagging data for easy retrieval and browsing.

[0167] In one embodiment, the data management subsystem of the present invention serves as the data center of the data service system. It interacts with all components of the system. Its underlying architecture is a highly available, distributed object storage service component built on MinIO, which stores data files of any type and provides data import, export, and management services. The system includes a data management module and a user login module.

[0168] ●User login module.

[0169] The user login module is used to verify the user's identity. It uses the same authentication system as the data annotation system, shares the same database and browser login information, so the annotation system can be used to annotate unlabeled data in the data management system.

[0170] ●User data management module.

[0171] The data management module is used to manage the data within the system, including data upload, download, deletion, retrieval, and calling data generation services to expand data.

[0172] In the data management module, beyond calling the MinIO interface to access files, we extend the data retrieval function. Users can define tags for uploaded data, enabling data retrieval within the system by tag. The existing front-end interface is optimized to provide data visualization. This module also includes an embedded data generation function, which uses data within the data management system to generate additional data by calling the data generation service and storing the generated data within the data management system.

[0173] 5. Model deployment and scheduling services.

[0174] The Model Deployment and Scheduling service addresses the "use" portion of the data lifecycle, automating the deployment and training of deep learning models. Coupled with the data annotation subsystem, this service provides pre-labeling for the data annotation process and automatically trains itself based on user annotation results to improve performance. This service can also train existing models using data from the data management subsystem and supports one-click deployment of trained models, which can then be exposed through an API for user access. This service, coupled with the data annotation and data management subsystems, fully leverages the value of existing data.

[0175] The model deployment and scheduling service is designed to provide ready-made web applications for machine learning predictions in production. The solution for implementing this service includes three components: middleware, front-end components, and back-end components. The architecture diagram is shown in Figure 10. The front-end component is responsible for receiving model inference requests; the back-end component is implemented in Python and integrates the Pytorch framework for loading and scheduling models. When a user loads a model, the front-end component starts a separate thread to receive all inference requests for the model. At the same time, the back-end uses Pytorch to load the model and start an instance of the model; the middleware is a separate RESTful API server that receives user management requests for the model and is responsible for model registration, loading, scheduling, deletion, and other requests. When using this service to deploy and host models, its technical implementation steps are as follows:

[0176] 1. Model packaging

[0177] This service provides a model packaging instruction. By executing this instruction, users can compress the specified model file, model parameters, control files, and dependent files into a tarball. At the same time, based on the parameters entered by the user, such as model name and model version, a configuration file is generated and placed in the compressed package for subsequent model loading. Among them, the model file refers to the model structure file defined using Pytorch, the model parameters are the serialized files of the model, and the control file is the core. A class is defined in it, and Python methods with different functions are defined under the class, including data preprocessing, data inference, data post-processing, model training, and other methods. Users need to manually rewrite them according to their own models. This service will call these methods in sequence after the model is loaded, thus completing a data request-model inference-data return process.

[0178] 2. Service startup

[0179] When the user starts this service, it will start the middleware and front-end components on two ports according to the user configuration. The middleware is responsible for management requests such as model registration, loading, scheduling, and deletion, while the front-end component is responsible for model inference requests.

[0180] 3. Model loading

[0181] When the user passes the packaged tarball to this service middleware, a process is started in the back-end component to decompress the package, extract all files to a temporary folder, read the configuration file, and load related files according to the model file name, model parameter name, and control file name recorded in the configuration file to generate a model instance. At the same time, a thread is started in the front-end component according to the model name in the configuration file to register the model for subsequent management and scheduling. Each thread in the front-end component corresponds to a model instance process in the back-end component.

[0182] 4. Model Inference

[0183] When the front-end component of this service receives an input request, it routes the request to the corresponding model for inference based on the request's routing. This service's front-end component is responsible for processing the HTTP request, parsing the requested data, and applying preprocessing functions specified in the user-written control file to the input data to ensure it meets the model's input requirements. After preprocessing, the service feeds the processed data into the model for inference. The inference results are then fed into a data post-processing method, which returns the resulting data as the final result to the user.

[0184] 5. Model management and extension

[0185] This service implements model management and expansion functions through middleware. Through the intercommunication between middleware, front-end components, and back-end components, new models can be added or model instances can be reduced according to user requests to achieve model management and expansion.

[0186] In one embodiment, the present invention utilizes the proposed model deployment and scheduling service as the underlying layer. Through a secondary development interface, it provides automatic deployment of user-uploaded models, active and passive model learning and training, model scheduling, and inference capabilities. This service can be used independently to deploy models, or it can be integrated with an annotation system to provide pre-annotation for the annotation system, enabling human-machine collaborative intelligent annotation. The service architecture is shown in Figure 8 and includes: a data annotation interface, a model management interface, a model scheduling module, a model management module, a dataset storage module, a model storage module, an annotation result storage module, and a model scheduling module.

[0187] ●Data annotation interface.

[0188] The data annotation interface is used to provide a model annotation service interface, and supports calling the annotation service provided by the model in the form of a RESTful interface to implement functions such as data pre-annotation and quality inspection.

[0189] ●Model management interface.

[0190] The model management interface is used to provide a model management service interface, supporting the unified deployment and scheduling of trained annotation models in the form of a RESTful interface, and providing a model management interface, including operations such as online, offline, version upgrade, and capacity expansion.

[0191] ●Model scheduling module.

[0192] The model scheduling module provides model training and model inference functions. It supports the CLI command line to use the existing datasets and labeled data of the annotation system to perform full iterative training of the model. It also supports real-time model training through user annotations, thereby assisting user annotation and improving user annotation efficiency; and uses the model to infer existing data.

[0193] The model scheduling module provides two training methods for model training: passive full training and active learning iterative training.

[0194] Full training: For models that need to be trained, the system backend loads the user's model file, control file, and dependent files, and uses the user-provided training method and loads the existing dataset and labeled data of the annotation system for iterative training.

[0195] Active learning iterative training: In manual annotation scenarios using auxiliary models, real-time model training is required. This paper utilizes an active learning solution, which enables interaction between humans and the text annotation model. The text annotation model provides annotation suggestions to the annotator, and the annotator's annotation training updates the text annotation model. Using the active learning model, we can automatically annotate unlabeled data. The annotation results are stored in a Redis database and persisted in batches to the file system. During subsequent annotation processes, this data is passed back to the annotation model for further training and improved performance. The data flow diagram is shown in Figure 9.

[0196] ●Model management module.

[0197] The model management module is used to provide functions for uploading, deleting, and managing models. It supports the CLI command line to upload local Pytorch format model definition files, parameter files, control files, and dependency files to the server, and the system loads the model for automatic deployment. It also supports the deletion of existing annotated models in the system. It also supports model configuration, and implements functions such as model online, offline, version upgrades, and capacity expansion. The number of model instances can be set by capacity expansion. In high concurrency, the capacity can be expanded to generate more instances to meet traffic demand. In low concurrency, fewer instances can be set to save system resources. The system can also be set to automatically adjust the number of instances according to the current traffic.

[0198] This invention conveniently loads one or more models and automatically creates a prediction API powered by a scalable web server. The model server can also run code pre- and post-processing based on prediction requests. Furthermore, the model server provides features such as logging, monitoring, and security for maintenance.

[0199] The present invention provides users with a standardized control file writing solution. When users upload model files, parameter files, dependency files and control files, the system verifies whether the model can run normally, packages the tested model files and transmits them to the middleware of the model deployment service, which deploys and provides an API. The system interacts with the model through the API, transmits the data to be labeled to the model, and the API returns the labeling results.

[0200] In terms of service provision, this paper uses Flask as the server to provide external services through a web interface. It re-encapsulates the API for model deployment and scheduling services, supports a RESTful interface format and interacts with the model management module to schedule models to implement functions such as data labeling and data quality inspection. For time-consuming tasks, Redis is used to store message queues, and Celery is used to build an asynchronous task queue. The labeled data is persisted through this task queue.

[0201] This service enables intelligent annotation through human-machine collaboration with annotation systems, achieving efficient annotation. Furthermore, you can use the labeled data in your data management system or the annotation system to fully train in-system or custom PyTorch models, and deploy the service with one click, eliminating the complex steps of model development, data preprocessing, and manual deployment, thereby improving the efficiency of R&D personnel.

[0202] ●Data set storage module.

[0203] The dataset storage module allows users to upload and delete datasets, as well as perform operations such as labeling datasets. When users train models using data from a data management system, this module is not required; the service obtains data through the interface provided by the data management system.

[0204] ●Model storage module.

[0205] The model storage module is used to record the model information uploaded by users into the database to facilitate user management and scheduling.

[0206] ●Annotation result storage module.

[0207] The annotation result storage module is used to persistently store the annotation results sent back by the user during the annotation process.

[0208] Figure 11 is a data flow diagram of the data service method of the present invention. As shown in Figure 11, the data service method includes the following steps:

[0209] 1. The user uses the collection plug-in to complete data collection and store the data in the data management subsystem;

[0210] 2. The user creates a labeling task in the data labeling subsystem and configures the task template, team members, etc.

[0211] 3. The user pushes the collected data in the data management subsystem to the annotation task of the annotation system;

[0212] 4. The user completes the annotation in the annotation system. During the annotation process, the data annotation subsystem interacts with the model deployment and scheduling service to establish a human-machine collaborative annotation model.

[0213] 5. If the user believes that the amount of data is insufficient, the user can use the labeled data or the collected unlabeled data to generate additional data;

[0214] 6. If the user has a custom model, they can upload the model to the model deployment and scheduling service;

[0215] 7. Users can use the data in the data management subsystem to train existing models;

[0216] 8. Users can deploy trained models, or directly deploy models built through active learning during the annotation process.

[0217] To sum up, in order to cover the entire life cycle of data, provide comprehensive services for data, and efficiently complete the complete work from data collection to data use, the present invention proposes an integrated data service method and system that integrates collection, labeling, generation, management, and use. The present invention is applicable to various types of data. When facing structured web pages, it provides a data collection plug-in for data that cannot be collected in batches through crawlers, so as to realize accurate data collection and push the data to the annotation module after formatting; the annotation module provides a data annotation function, and controls the quality of data annotation with a strict annotation process. The annotation module and the data management module communicate with each other to realize the push from the data management module to the annotation task, and transmit the annotation results back to the data management module for storage; the data generation module and the data management module communicate with each other to realize the use of data stored in the data management module to generate forged and adversarial data in various scenarios, thereby achieving the purpose of expanding data; the data usage module contains a model deployment and scheduling function to realize the automatic deployment and scheduling of the model, and provides pre-labeling and model active learning functions, combined with the annotation module to realize the human-computer collaborative annotation mode, which greatly improves the annotation efficiency. In addition, the module provides the function of using the data in the data management module to train the internal model, and supports the deployment of the trained model online, providing services to the outside world, simplifying the work of model training and deployment, and greatly reducing the time cost from basic data to model implementation.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data service method, characterized in that: The method comprises: Determine the raw data that needs to be captured through user behavior, and combine it with the data format determined by the annotation task to obtain the original data set; Generating data based on the original data set to obtain a generated data set; wherein the data format in the generated data set satisfies the data format determined by the labeling task; In combination with the labeling task configured by the user, the trained data labeling model labels the data in the original data set and the generated data set to obtain a labeled data set; After the data annotation model, original data set, generated data set and annotated data set are stored in the data center, the permissions and retrieval tags of the data annotation model or data are set; wherein the data center is provided with a data annotation interface that provides model annotation services and a model management interface that provides model management services.

2. The method according to claim 1, characterized in that The original data to be captured is determined by user behavior, and combined with the data format determined by the labeling task to obtain the original data set, including: Obtaining a labeling task created by a system administrator, wherein the labeling task includes a data format and a labeling user corresponding to the labeling task, wherein the data format includes: a data type and a number of data items; When a labeling user logs in, the user identity corresponding to the labeling user is verified and all labeling tasks corresponding to the labeling user are obtained; Based on the annotation task selected by the annotation user, the data type and number of data items collected are limited; Monitor user behavior and capture web page areas and content by limiting the type and number of data collected to obtain the original data set.

3. The method according to claim 2, characterized in that The monitoring of user behavior and capturing of web page areas and content in combination with limiting the data types and number of data items collected to obtain the original data set include: When the data type is text data and the number of data items is a single item, monitoring whether a corresponding shortcut key is pressed after the user uses the mouse to select a section of text content, and capturing the text content selected by the mouse when the corresponding shortcut key is pressed; When the data type is image data and the number of data items is a single item, monitoring whether a preset shortcut key is pressed, and when the preset shortcut key is pressed, capturing a picture of the image area where the cursor is located or starting a screenshot function; When the number of data items is in batches, the HTML source code of the current page is obtained, and when the corresponding shortcut key is pressed, the xpath of the element in the cursor area in the webpage DOM is analyzed, and the webpage is re-rendered to highlight the element node content with the same xpath.

4. The method according to claim 1, characterized in that The data generation process includes: Based on the data falsification tool, data enhancement is performed on the data in the original data set; and, Based on the data adversarial generation tool, adversarial samples of the data in the original data set are generated.

5. The method according to claim 1, characterized in that The trained data annotation model annotates the data in the original data set and the generated data set in combination with the annotation task configured by the user to obtain an annotated data set, including: In combination with the annotation task configured by the user, the large model is called to implement data pre-annotation; wherein the large model includes: the online large model ChatGPT or the locally deployed large model ChatGLM; After the annotation user checks the pre-annotation results, the data annotation model is trained based on the checked results to obtain a trained data annotation model; The data in the original data set and the generated data set are labeled based on the trained data labeling model to obtain a labeled data set.

6. The method according to claim 5, characterized in that The method combines the labeling task configured by the user and calls the large model to implement data pre-labeling, including: Obtaining annotation configuration information input by a user; wherein the annotation configuration information is used to clarify the data and task type to be annotated, as well as various attributes involved in the annotation process; Based on the annotation configuration information, fill the data to be annotated into a predefined prompt template to generate a prompt corresponding to the task; The prompt is translated into several languages, and a piece of data is extracted from the data manually annotated by the user as a sample and then translated into the corresponding language; After the sample of each language is embedded in the prompt, it is input to the big model as the guide information, so that the big model receiving the guide information annotates the data to be annotated based on the prompt of the corresponding language, and obtains the annotation result of the corresponding language after processing the redundant data in the result returned by the big model; Translate the annotation results in the corresponding language into the language of the data to be annotated, and obtain a number of answers in the same language as the data to be annotated; The multiple answers with the same language as the data to be annotated and the prompt with the same language as the data to be annotated are input into the large model, so that the large model selects one answer from the multiple answers with the same language as the data to be annotated as the annotation result of the data to be annotated.

7. The method according to claim 6, characterized in that After the large model selects one answer from the plurality of answers in the same language as the data to be annotated as the annotated result of the data to be annotated, the large model further includes: Display the annotation results of the data to be annotated to the user so that the user can make revisions; Add the revised annotation results to the next prompt to annotate the next data to be annotated.

8. The method according to claim 1, characterized in that The process of providing model management services based on the model management interface includes: Based on the management request received by the model management interface, the packaged tar compressed package is passed to the middleware; wherein the packaged tar compressed package is compressed based on the model file, model parameters, control file, dependency file and configuration file specified by the management request, the model file refers to the model structure file defined by Pytorch, the model parameter is the serialized file of the model, the control file defines a class containing Python methods with different functions, and the configuration file is generated according to the model file name, model parameter name and model version input by the user; A process is started in the backend component to decompress the packaged tarball and extract all files into a temporary folder; The back-end component reads the configuration file, and loads related files according to the model file name, model parameter name and control file name recorded in the configuration file to generate a model instance; The backend component starts a thread in the frontend component according to the model file name recorded in the configuration file to register the model; each thread in the frontend component corresponds to a model instance process in the backend component; When the front-end component receives a model inference request, it parses the model inference request to obtain input data; Processing the input data according to the preprocessing function specified in the control file to obtain preprocessed data; Input the preprocessed data into the model for inference, and perform data post-processing on the inference results; The data is post-processed and returned to the user.

9. The method according to claim 8, characterized in that The method also includes: training the model based on passive full-scale training and / or active learning iterative training.

10. A data service system, characterized in that: The system comprises: Data collection tools are used to determine the raw data that needs to be captured through user behavior and combine it with the data format determined by the annotation task to obtain the original data set; A data generation service, used to generate data based on the original data set to obtain a generated data set; wherein the data format in the generated data set satisfies the data format determined by the annotation task; A data annotation subsystem, used to annotate the data in the original data set and the generated data set by the trained data annotation model in combination with the annotation task configured by the user, so as to obtain an annotated data set; The data management subsystem is used to set permissions and retrieval tags for the data annotation model or data after storing the data annotation model, original data set, generated data set and annotated data set in the data center; wherein the data center is provided with a data annotation interface that provides model annotation services and a model management interface that provides model management services.

Images