Data processing method, device and equipment

Abstract

The embodiment of the invention provides a data processing method, device and equipment, and the method comprises the steps: receiving a labeling request for to-be-labeled data; in response to the annotation request, processing the to-be-annotated data by using a multi-modal large language model, and determining a first annotation result for the to-be-annotated data; splitting the annotation demand information to obtain a plurality of pieces of sub-demand information, and determining a data processing model corresponding to each piece of sub-demand information according to an auxiliary annotation rule corresponding to each piece of sub-demand information; according to each data processing model, performing identification processing on information corresponding to the auxiliary labeling rule in the first labeling result to obtain an auxiliary labeling result corresponding to each piece of sub-demand information; and according to the auxiliary annotation result corresponding to each piece of sub-demand information and the first annotation result, determining a target annotation result for the to-be-annotated data.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular, to a data processing method, apparatus, and device. Background Art

[0002] To protect user privacy and ensure data security, business data, user data, and other data can be detected through a detection model. Therefore, data labels can be determined through data annotation to train the detection model with the annotated data. Among them, data annotation refers to the process of adding structured information to the original data, and the added annotation data can be used to assist in understanding the data characteristics of the original data. For example, annotation data can be added to the original data through manual annotation.

[0003] However, as the amount of data to be annotated increases and the data structure becomes more complex, the annotation efficiency and accuracy of manual annotation are low. Therefore, the embodiments of this specification provide a better technical solution to improve data annotation efficiency and accuracy. Summary of the Invention

[0004] The purpose of the embodiments of this specification is to provide a better technical solution to improve data annotation efficiency and accuracy.

[0005] To achieve the above technical solution, the embodiments of this specification are implemented as follows: A data processing method provided by the embodiments of this specification, the method includes: receiving an annotation request for data to be annotated; in response to the annotation request, using a multimodal large language model to process the data to be annotated to determine a first annotation result for the data to be annotated; splitting the annotation requirement information to obtain multiple sub-requirement information, and determining a data processing model corresponding to each sub-requirement information according to the auxiliary annotation rule corresponding to each sub-requirement information; identifying the information corresponding to the auxiliary annotation rule in the first annotation result according to each data processing model to obtain an auxiliary annotation result corresponding to each sub-requirement information; and determining a target annotation result for the data to be annotated according to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result.

[0006] A data processing device provided by an embodiment of this specification, the device includes: a request receiving module, configured to receive a labeling request for data to be labeled, where the labeling request includes labeling requirement information corresponding to the data to be labeled, and the data to be labeled includes multi-modal data; a model processing module, configured to, in response to the labeling request, use a multi-modal large language model to process the data to be labeled and determine a first labeling result for the data to be labeled; a model determination module, configured to perform splitting processing on the labeling requirement information to obtain a plurality of sub-requirement information, and determine a data processing model corresponding to each sub-requirement information according to the auxiliary labeling rule corresponding to each sub-requirement information; an information recognition module, configured to, according to each data processing model, perform recognition processing on the information corresponding to the auxiliary labeling rule in the first labeling result to obtain an auxiliary labeling result corresponding to each sub-requirement information; a result determination module, configured to determine a target labeling result for the data to be labeled according to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result.

[0007] A data processing device provided by an embodiment of this specification, the data processing device includes: a processor; and a memory arranged to store computer-executable instructions, the executable instructions, when executed, cause the processor to: receive a labeling request for data to be labeled; in response to the labeling request, use a multi-modal large language model to process the data to be labeled and determine a first labeling result for the data to be labeled; perform splitting processing on the labeling requirement information to obtain a plurality of sub-requirement information, and determine a data processing model corresponding to each sub-requirement information according to the auxiliary labeling rule corresponding to each sub-requirement information; according to each data processing model, perform recognition processing on the information corresponding to the auxiliary labeling rule in the first labeling result to obtain an auxiliary labeling result corresponding to each sub-requirement information; determine a target labeling result for the data to be labeled according to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result.

[0008] An embodiment of this specification also provides a storage medium for storing computer-executable instructions. When the executable instructions are executed by a processor, the following process is implemented: receiving a labeling request for data to be labeled; in response to the labeling request, using a multimodal large language model to process the data to be labeled and determining a first labeling result for the data to be labeled; splitting the labeling requirement information to obtain multiple sub-requirement information, and determining a data processing model corresponding to each sub-requirement information according to the auxiliary labeling rule corresponding to each sub-requirement information; according to each data processing model, performing identification processing on the information corresponding to the auxiliary labeling rule in the first labeling result to obtain an auxiliary labeling result corresponding to each sub-requirement information; and determining a target labeling result for the data to be labeled according to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result.

[0009] An embodiment of this specification also provides a computer program product, including a computer program. When the computer program is executed by a processor, the following process is implemented: receiving a labeling request for data to be labeled; in response to the labeling request, using a multimodal large language model to process the data to be labeled and determining a first labeling result for the data to be labeled; splitting the labeling requirement information to obtain multiple sub-requirement information, and determining a data processing model corresponding to each sub-requirement information according to the auxiliary labeling rule corresponding to each sub-requirement information; according to each data processing model, performing identification processing on the information corresponding to the auxiliary labeling rule in the first labeling result to obtain an auxiliary labeling result corresponding to each sub-requirement information; and determining a target labeling result for the data to be labeled according to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result. Description of the Drawings

[0010] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in this specification. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts. Figure 1 This is an embodiment of a data processing method in this specification; Figure 2 This is another embodiment of a data processing method in this specification; Figure 3 This is a schematic diagram of a data rendering result in this specification; Figure 4 This is a schematic diagram of a data display page in this specification; Figure 5 It is a schematic diagram of an auxiliary tool library in this specification; Figure 6 It is a schematic diagram of a triggering method for an interactive retrieval request in this specification; Figure 7 It is a schematic diagram of a data processing process in this specification; Figure 8 It is an embodiment of a data processing device in this specification; Figure 9 It is an embodiment of a data processing device in this specification. Detailed implementation manners

[0011] The embodiments of this specification provide a data processing method, device and equipment.

[0012] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

[0013] The embodiments of this specification provide a better technical solution for improving data annotation efficiency and annotation accuracy. Data annotation is a process of adding structured information to raw data. The added annotation data can be used to assist in understanding the data characteristics of the raw data, so as to improve the data processing efficiency of subsequent raw data. For example, annotation data can be added to the raw data through manual annotation. However, as the amount of data to be annotated increases and the data structure becomes more complex, the annotation efficiency and annotation accuracy of manual annotation are low. Therefore, the embodiments of this specification provide a better technical solution for improving data annotation efficiency and annotation accuracy. In this solution, by receiving an annotation request for the data to be annotated, the annotation request includes annotation requirement information corresponding to the data to be annotated, and the data to be annotated includes multimodal data. In response to the annotation request, a multimodal large language model is used to process the data to be annotated, and a first annotation result for the data to be annotated is determined. The annotation requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary annotation rules corresponding to each sub-requirement information, a data processing model corresponding to each sub-requirement information is determined. According to each data processing model, the information corresponding to the auxiliary annotation rules in the first annotation result is identified to obtain an auxiliary annotation result corresponding to each sub-requirement information. According to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result, a target annotation result for the data to be annotated is determined. In this way, in the case where the data to be annotated and the annotation requirements are relatively complex, on the one hand, the first annotation result can be quickly and accurately determined through the understanding, summarization, and reasoning capabilities of the large language model. On the other hand, the information corresponding to the auxiliary annotation rules in the first annotation result can be identified according to the data processing model determined according to the auxiliary annotation rules corresponding to each sub-requirement information to obtain the auxiliary annotation result. Furthermore, the target annotation result for the data to be annotated can be determined according to the first annotation result and the auxiliary annotation result that can meet the sub-requirement information, improving the annotation efficiency and standard accuracy of data annotation. The specific processing can refer to the specific content in the following embodiments.

[0014] As Figure 1 shown, the embodiments of this specification provide a data processing method. The execution subject of this method can be a server. The server can be an independent server or a server cluster composed of multiple servers, etc. The server can be a background server such as a financial business or an online shopping business, or a background server of a certain application program, etc. In this embodiment, the execution subject is taken as an example of a server for detailed description. The method can specifically include the following steps: In step S102, an annotation request for the data to be annotated is received.

[0015] Among them, the annotation request includes annotation requirement information corresponding to the data to be annotated. The data to be annotated includes multimodal data, and the data to be annotated can be any multimodal data. For example, the data to be annotated can include data such as image data, text data, voice data, video data, and point cloud data. The annotation requirement information can include annotation specifications, annotation rules, annotation strategies, etc. for the data to be annotated.

[0016] In implementation, the server can receive an annotation request triggered by a user for one or more data to be annotated. Alternatively, the server can also trigger an annotation request for the data to be annotated obtained within the preset annotation period based on the preset annotation period. Or, the server can also receive an annotation request for sample data used to train the service processing model. The sample data can be used as the data to be annotated, and the service processing model can be a model for processing a preset service constructed based on a preset machine learning algorithm.

[0017] The above method for obtaining the annotation request of the data to be annotated is an optional and implementable obtaining method. In actual application scenarios, there can also be multiple different obtaining methods, and different obtaining methods can be selected according to different actual application scenarios. The embodiments of this specification do not make specific limitations on this.

[0018] In addition, the annotation requirement information corresponding to different types of data in the data to be annotated can be different. For example, taking the data to be annotated including image data, text data, voice data, video data, and point cloud data as an example, the annotation requirement information for different data in the data to be annotated can be as shown in Table 1 below.

[0019] Table 1

[0020] Among them, the object detection requirement can be to identify and label the positions of specific objects in an image, usually using bounding boxes. The semantic segmentation requirement can be to classify each pixel of the image and label different regions or objects. The instance segmentation requirement can be to distinguish different instances within the same category. The key point detection requirement can be to label specific key points on an object, such as the key parts of a face. The sentiment analysis requirement can be to label the sentiment tendency of the text, such as positive, negative, or neutral. The named entity recognition requirement can be to label entities such as person names, place names, and organization names in the text. The relation extraction requirement can be to label the relationships between entities in the text. The text classification requirement can be to classify the text into predefined categories. The speech-to-text requirement can be to convert speech data into text. The audio event annotation requirement can be to label specific events or sound types in the audio. The frame-level annotation requirement can be to perform image annotation on each frame of the video. The action recognition requirement can be to label the actions or movements of people in the video. The time series annotation requirement can be to label the time points when events occur. The point cloud object detection requirement can be to identify points with significant geometric features from unordered point cloud data to describe the local or global features of an object. The point cloud key point annotation requirement can be that key points usually contain key information about the shape, structure, or function of an object. The point cloud semantic segmentation requirement can be that in the application scenario of an autonomous vehicle, the points in the point cloud can be classified as roads, vehicles, pedestrians, buildings, vegetation, etc. The multi-dimensional fusion annotation requirement can be to simultaneously annotate the image data collected from sensors in dimensions such as 2D and 3D and establish connections.

[0021] As shown in Table 1 above, the annotation requirement information corresponding to different types of data included in the data to be annotated can be different. At the same time, the annotation requirement information corresponding to the same type of data included in the data to be annotated can be the same or different. For example, the annotation requirement information for image data 1 can include classification requirements and object detection requirements, and the annotation requirement information for image data 2 can be component requirements, speech segmentation requirements, and instance segmentation requirements.

[0022] The annotation requirement information corresponding to the data to be annotated can select different annotation requirements according to different actual application scenarios, and this embodiment of the present specification does not make specific limitations on this.

[0023] In step S104, in response to the annotation request, the multi-modal large language model is used to process the data to be annotated to determine the first annotation result for the data to be annotated.

[0024] Among them, a Multimodal Large Language Model (MMLLM) can be an advanced Natural Language Processing (NLP) model based on deep learning technology for processing and generating various modal data. Its underlying transformer is a set of neural networks, which can consist of an encoder and a decoder with self-attention functions. It can understand and generate human language by processing a large amount of text data, can process and generate high-dimensional data, and thus perform various natural language processing tasks. Due to the large number of parameters and extensive training datasets of large language models, large language models can capture the richness and nuances of language expressions and can display the captured information through the powerful computing power of billions to trillions of parameters, and have powerful prediction and analysis capabilities in multiple fields such as language understanding and image recognition.

[0025] In implementation, the server can input the prompt information corresponding to the annotation requirement information and the data to be annotated into the multimodal large language model to obtain the first annotation result for the data to be annotated.

[0026] For example, taking the data to be annotated as image data and text data as an example, the prompt information corresponding to the annotation requirement information can be: "According to the image data and text data, determine the classification information of the detection object corresponding to the image data and text data". The server can input this prompt information and the above image data and text data into the multimodal large language model to obtain the first annotation result for the data to be annotated, and the obtained first annotation result can be: "The classification information corresponding to this detection can be 'foreign language book category'."

[0027] The above method for determining the first annotation result is an optional and implementable determination method. In actual application scenarios, there can also be various different determination methods, and different determination methods can be selected according to the different actual application scenarios. This embodiment of the specification does not make specific limitations on this.

[0028] In step S_{106}, the annotation requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary annotation rules corresponding to each sub-requirement information, the data processing models corresponding to each sub-requirement information are determined.

[0029] Among them, the auxiliary annotation rule can be a rule for determining the auxiliary information corresponding to the annotation requirement in the annotation result. For example, taking the annotation requirement as the sentiment analysis requirement for text data as an example, the auxiliary annotation rule can be to obtain and annotate the lexical analysis knowledge related to the preset emotion keywords in the text data, etc.

[0030] In implementation, the server can split and process the annotation requirement information according to the requirement information for the data of each modality in the annotation requirement information, to obtain multiple sub-requirement information. For example, assuming that the annotation requirement information includes a classification requirement for image data and a speech-to-text requirement for speech data, then the server can split the annotation requirement information into sub-requirement information 1 corresponding to the classification requirement and sub-requirement information 2 corresponding to the speech-to-text requirement.

[0031] In addition, the methods for splitting and processing the annotation requirement information can be diverse. For example, the server can also split and process the annotation requirement information according to a pre-trained splitting model to obtain multiple sub-requirement information, etc. Among them, the splitting model can be a model constructed based on a preset machine learning algorithm, and different splitting and processing methods can be selected according to different actual application scenarios. This specification does not make specific limitations on this.

[0032] The server can determine a data processing model corresponding to each sub-requirement information according to the auxiliary annotation rule corresponding to each sub-requirement information. For example, taking the above auxiliary annotation rule as obtaining and annotating the lexical parsing knowledge related to preset emotion keywords in text data as an example, the corresponding data processing model can include an emotion keyword recognition model and a keyword parsing model, etc.

[0033] In step S108, according to each data processing model, identify and process the information in the first annotation result corresponding to the auxiliary annotation rule, to obtain an auxiliary annotation result corresponding to each sub-requirement information.

[0034] In step S110, determine the target annotation result for the data to be annotated according to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result.

[0035] In implementation, the server can perform information fusion processing on the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result to obtain the target annotation result for the data to be annotated. For example, the server can use the powerful understanding ability of the large language model to perform information fusion processing on the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result to obtain the target annotation result for the data to be annotated.

[0036] Alternatively, the server can also send the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result to a preset management party, and receive the target annotation result for the data to be annotated determined by the preset management party according to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result.

[0037] In addition, there can be various methods for determining the target annotation result, and different determination methods can be selected according to different actual application scenarios. The embodiments of this specification do not make specific limitations on this.

[0038] After performing annotation processing on the data to be annotated (i.e., obtaining the target annotation result for the data to be annotated), the server can use the data to be annotated as sample data, and train a business processing model based on the data to be annotated and the corresponding target annotation result, so as to process a preset business through the trained business processing model.

[0039] The embodiments of this specification provide a data processing method. By receiving an annotation request for the data to be annotated, the annotation request includes annotation requirement information corresponding to the data to be annotated, and the data to be annotated includes multi-modal data. In response to the annotation request, using a multi-modal large language model, the data to be annotated is processed to determine a first annotation result for the data to be annotated. The annotation requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary annotation rules corresponding to each sub-requirement information, data processing models corresponding to each sub-requirement information are determined. According to each data processing model, the information corresponding to the auxiliary annotation rules in the first annotation result is identified to obtain auxiliary annotation results corresponding to each sub-requirement information. According to the auxiliary annotation results corresponding to each sub-requirement information and the first annotation result, the target annotation result for the data to be annotated is determined. In this way, in the case where the data to be annotated and the annotation requirements are relatively complex, on the one hand, through the understanding, summarization, and reasoning capabilities of the large language model, the first annotation result can be quickly and accurately determined. On the other hand, according to the data processing models determined by the auxiliary annotation rules corresponding to each sub-requirement information, the information corresponding to the auxiliary annotation rules in the first annotation result is identified to obtain the auxiliary annotation result. Furthermore, according to the first annotation result and the auxiliary annotation result that can meet the sub-requirement information, the target annotation result for the data to be annotated can be determined, improving the annotation efficiency and standard accuracy of data annotation.

[0040] In practical applications, the data processing model can include a duplicate recognition model, a relevance recognition model, and a knowledge assistance model. Among them, the duplicate recognition model can be used to recognize duplicate information in the annotation result, the relevance recognition model can be used to recognize information related to preset keywords in the annotation result, and the knowledge assistance model can be used to obtain knowledge related to the preset keywords in the annotation result.

[0041] In practical applications, the specific processing method for determining the target annotation result for the data to be annotated according to the auxiliary annotation results corresponding to each sub-requirement information and the first annotation result in step S110 can be various. The following provides an optional processing method, such as Figure 2As shown, it may specifically include the processing of the following steps S1102 to S1104.

[0042] In step S1102, obtain the data rendering method corresponding to each auxiliary annotation rule.

[0043] Among them, the data rendering method may include a highlighting rendering method, an underlining rendering method, a hovering rendering method, etc.

[0044] In implementation, the server can preset different data rendering methods according to each different auxiliary annotation rule. For example, for auxiliary annotation rule 1 (such as marking out orientation words), the corresponding data rendering method can be a method of rendering through yellow highlighting; for auxiliary annotation rule 2 (such as marking out color-related words), the corresponding data rendering method can be a method of rendering through blue highlighting; for auxiliary annotation rule 3 (such as marking out OCR recognition results), the corresponding data rendering method can be a method of rendering through red highlighting, etc.

[0045] In step S1104, according to the data rendering method, perform rendering processing on the auxiliary annotation results corresponding to each sub-requirement information respectively, and determine the target annotation result for the data to be annotated based on the rendered auxiliary annotation results and the first annotation result.

[0046] In implementation, the server can fuse the information of the rendered auxiliary annotation results and the first annotation result to obtain the target annotation result. For example, the obtained target annotation result can be as Figure 3 shown, where the data rendered through yellow highlighting in the target annotation result can be the rendered auxiliary annotation result corresponding to the above-mentioned auxiliary annotation rule 1, the data rendered through blue highlighting can be the rendered auxiliary annotation result corresponding to the above-mentioned auxiliary annotation rule 2, and the data rendered through red highlighting can be the rendered auxiliary annotation result corresponding to the above-mentioned auxiliary annotation rule 3.

[0047] In addition, the server can also send the result obtained by fusing the rendered auxiliary annotation results and the first annotation result to a preset management party. In this way, the preset management party can, with the assistance of the auxiliary annotation results with different rendering effects, determine the target annotation result for the data to be annotated based on the first annotation result.

[0048] In addition, the server can also send the rendered auxiliary annotation results corresponding to different sub-requirement information to the preset management party. In this way, the preset management party can select the auxiliary annotation results that need to be displayed from multiple auxiliary annotation results. For example, as Figure 4As shown, in the display page of the preset management party, selection boxes corresponding to different auxiliary annotation rules can be displayed, such as the "Global Display" selection box, the "Position Highlighting" selection box, and the "OCR Recognition" selection box. The preset management party can select the auxiliary annotation results to be displayed in the fused result by inputting selection instructions on the selection boxes, and then determine the target annotation results based on the display results.

[0049] The server can select and uniformly call the adapted annotation auxiliary user-defined functions (UDFs) from the auxiliary tool library according to the annotation task requirements (i.e., annotation requirement information), and perform multi-layer rendering according to the annotation requirement information to produce the data after rendering with multi-auxiliary annotation capabilities (i.e., the rendered auxiliary annotation results). In this way, when the preset management party executes the annotation task, it can directly select the auxiliary annotation capabilities to be displayed on the annotation page according to the pre-rendered classification (i.e., the auxiliary annotation rules). In this way, through the parallel invocation of multi-auxiliary annotation capabilities, combining the actual demands of the annotation task to call and assemble the auxiliary tool library, and supporting the corresponding rendering scheme, it can not only enable the annotation task to simultaneously call multiple capabilities, but also select different rendering schemes on the annotation display page to use the auxiliary annotation capabilities differently, greatly reducing the mutual interference between the auxiliary annotation capabilities and enhancing the autonomy of user use.

[0050] In practical applications, the specific processing methods for determining the data processing models corresponding to each sub-requirement information according to the auxiliary annotation rules corresponding to each sub-requirement information in step S106 above can be various. The following provides an optional processing method, which can specifically include the processing in step S1062.

[0051] In step S1062, the annotation requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary annotation rules corresponding to each sub-requirement information, the data processing models corresponding to each sub-requirement information are screened out from the pre-constructed auxiliary tool library.

[0052] Among them, the call protocols corresponding to the call interfaces of different data processing models in the auxiliary tool library are the same.

[0053] In implementation, due to the increasing complexity of the data to be annotated and the annotation requirement information, to improve the annotation efficiency and annotation accuracy, an auxiliary tool library can be pre-constructed. Among them, as Figure 5 shown, the auxiliary tool library can have the following characteristics: 1. Unified capability integration: For multimodal data such as image data, text data, audio data, video data, etc., as well as the auxiliary annotation capabilities involved in various knowledge bases, unified integration and encapsulation have been carried out, which can provide an engineering foundation for the reusability of subsequent unified development. For example, in the auxiliary tool library as shown in Figure 5 , retrieval highlighting algorithms, rewriting correction algorithms, summary Q&A algorithms, and knowledge hint algorithms for processing text data can be integrated. Optical Character Recognition (OCR) technology, element understanding algorithms, graphic-text matching algorithms, and summary Q&A algorithms for processing image data. Automatic Speech Recognition (ASR) technology, audio segmentation algorithms, audio caption services, and voiceprint recognition technology for processing audio data. Key frame recognition algorithms, video understanding algorithms, video segmentation algorithms, and topic tracking algorithms for processing video data. In addition, multiple industry knowledge bases can also be integrated into the auxiliary tool library, such as entity knowledge bases (such as risk entity knowledge, financial product knowledge, government and enterprise service knowledge, and medical disease knowledge, etc.), Q&A knowledge (such as security risk knowledge, insurance and wealth management knowledge, assistant Q&A knowledge, life services, etc.), evaluation knowledge (such as base capabilities, security identification, financial compliance, medical health, etc.), and multimodal knowledge (such as content security knowledge, graphic-text understanding knowledge, multimodal certificate knowledge, and voice intention knowledge, etc.); 2. Unified service integration: Integrate the algorithm services, retrieval services, and knowledge management involved in the auxiliary annotation capabilities for convenient subsequent use and support subsequent expansion and addition; 3. Unified interface call: By uniformly defining the service call protocol, a paradigm is provided for large-scale applications, which can significantly reduce subsequent maintenance and understanding costs and facilitate standardized operation and maintenance upgrades.

[0054] The server can integrate the above capabilities and services, deploy them in a toolized manner, and build an auxiliary tool library capable of performing knowledge retrieval services and knowledge standard management.

[0055] In this way, through unified annotation capabilities, unified services, and unified interface calls, high reusability and expandability can be provided for subsequent single-task auxiliary development, significantly reducing development costs.

[0056] In practical applications, the auxiliary tool library can contain multiple data conversion models for processing multimodal data. In step S1108, according to each data processing model, the information corresponding to the auxiliary annotation rules in the first annotation result can be identified and processed, and the specific processing methods for obtaining the auxiliary annotation results corresponding to each sub-requirement information can be various. The following provides an optional processing method, such asFigure 2 As shown, it may specifically include the processing of the following steps S1082 to S1086.

[0057] In step S1082, determine the sub-annotation data corresponding to each sub-requirement information in the data to be annotated.

[0058] In implementation, the server can use a large language model to perform intent recognition processing on the sub-requirement information, so as to determine the sub-annotation data corresponding to each sub-requirement information in the data to be annotated according to the obtained intent recognition result.

[0059] In step S1084, use the data conversion model in the auxiliary tool library to perform text conversion processing on the sub-annotation data to obtain the converted sub-annotation data.

[0060] Among them, the data conversion model can be a model constructed based on a preset machine learning algorithm for performing text conversion processing on non-text data.

[0061] In implementation, the server can call the data conversion model corresponding to the data type from the auxiliary tool library according to the data type of the non-text data in the sub-annotation data, and perform text conversion processing on the non-text data in the sub-annotation data according to the called data conversion model to obtain the converted sub-annotation data.

[0062] In step S1086, according to each data processing model, based on the converted sub-annotation data and the first annotation result, perform recognition processing on the information corresponding to the auxiliary annotation rule in the first annotation result to obtain the auxiliary annotation result corresponding to each sub-requirement information.

[0063] In implementation, since the amount of information contained in the converted sub-data is relatively large, therefore, the server can combine the converted sub-annotation data to perform recognition processing on the information corresponding to the auxiliary annotation rule in the first annotation result to improve the determination accuracy of the auxiliary annotation result.

[0064] For example, taking the data processing model as a knowledge assistance model as an example, the server can call the knowledge assistance model to respectively obtain the knowledge related to the preset keywords in the converted sub-data and the knowledge related to the preset keywords in the first annotation result, and then determine the auxiliary annotation result corresponding to the sub-requirement information according to the two pieces of knowledge obtained.

[0065] In practical applications, the specific processing method for determining the target annotation result for the data to be annotated according to the auxiliary annotation result and the first annotation result corresponding to each sub-requirement information can be various. The following provides an optional processing method, as Figure 2 shown, it may specifically include the processing of the following steps S1106 to S11010.

[0066] In step S1106 , an interactive search request for the target keyword in the first annotation result is received.

[0067] In implementation, since the first annotation result may contain words that are difficult to understand, such as highly professional words and ambiguous words, the server may receive an interactive search request for target keywords in the first annotation result.

[0068] There are many ways to trigger the interactive search request, for example, Figure 6 As shown, the user can trigger an interactive search request for the target keyword by sliding the word, or the user can also enter an interactive search request for the target keyword in the interactive input box. In addition, there can be a variety of different triggering methods, which may vary according to the actual application scenario. The embodiments of this specification do not make specific limitations on this.

[0069] In step S1108 , in response to the interactive search request, target knowledge corresponding to the target keyword is determined.

[0070] In practical applications, there may be various specific processing methods for determining the target knowledge corresponding to the target keyword in step S1108. An optional processing method is provided below, which may specifically include the processing of the following steps A1 to A2.

[0071] In step A1, the interactive search request is processed for intent recognition to obtain an intent recognition result.

[0072] In implementation, the server can use a large language model or a pre-trained intent recognition model to perform intent recognition processing on the search content corresponding to the interactive retrieval request input by the user to obtain an intent recognition result, wherein the intent recognition model can be a model built based on a preset machine learning algorithm.

[0073] In step A2, the large language model is used to determine the target knowledge corresponding to the target keyword based on the intent recognition result and the target keyword.

[0074] In practice, the server can leverage the analytical and comprehension capabilities of the large language model to determine the target knowledge corresponding to the target keyword based on the intent recognition results and the target keyword. This allows the server to accurately determine the target knowledge corresponding to the target keyword through the large language comprehension capabilities, even when the user's search query is semantically ambiguous, thereby improving the accuracy of knowledge determination.

[0075] In addition, there are various specific processing methods for determining the target knowledge corresponding to the target keyword in the above step S1108. An optional processing method is provided below, which may specifically include the processing of the following steps B1 to B2.

[0076] In step B1, according to the interactive retrieval request, determine the target knowledge base corresponding to the interactive retrieval request.

[0077] In implementation, as Figure 5 shown, the pre-constructed auxiliary tool library can also contain multiple knowledge bases, and the target knowledge base corresponding to the interactive retrieval request can be screened out from the pre-constructed auxiliary tool library according to the interactive retrieval request.

[0078] In step B2, according to the similarity between the target keyword and each piece of knowledge stored in the target knowledge base, determine the target knowledge corresponding to the target keyword among the knowledge stored in the target knowledge base.

[0079] In implementation, the server can determine the similarity between the target keyword and each piece of knowledge stored in the target knowledge base based on a preset similarity algorithm (such as the cosine algorithm, Euclidean distance algorithm, etc.), and then screen out the target knowledge corresponding to the target keyword according to the similarity.

[0080] Through the unified access of the auxiliary tool library, the auxiliary demands of the annotators in the face of task emergencies can be solved. For example, the annotator can select the target keyword by swiping down and trigger interactive retrieval requests such as content retrieval, translation, and error correction for the target keyword. The server can uniformly return the obtained target knowledge after calling through the interface.

[0081] Alternatively, the annotator can also send an interactive retrieval instruction to the auxiliary tool library in the way of instruction delivery, and then, through the algorithm model integrated in the auxiliary tool library, determine and return the target knowledge according to the intention recognition result.

[0082] In this way, through the above two methods, it can enable the annotator to select and call the appropriate auxiliary annotation capabilities in the multi-element and extensible ability pool (such as knowledge base retrieval, directional retrieval, public network retrieval, OCR ability, Chinese-English translation ability, and text error correction ability) according to the actual emergency demands during the annotation process, which can improve the extensibility of the auxiliary annotation ability and the annotation efficiency and quality.

[0083] In step S11010, according to the target knowledge, the auxiliary annotation result corresponding to each sub-demand information, and the first annotation result, determine the target annotation result for the data to be annotated.

[0084] In implementation, the server can correct the first annotation result based on the target knowledge, the auxiliary annotation result corresponding to each sub-demand information, and the first annotation result according to the pre-trained result correction model, and obtain the target annotation result for the data to be annotated, where the result correction model can be a model constructed according to a preset machine learning algorithm.

[0085] In this way, as Figure 7 shown, by pre-calling a data processing model to determine multiple auxiliary annotation results, and then determining the target knowledge based on the external temporary adjustment of capabilities, the target annotation result can be assisted to determine the target annotation result, improving the accuracy of determining the target annotation result.

[0086] In practical applications, after determining the target annotation result of the data to be annotated, the data to be annotated can also be processed for risk review according to the target annotation result. The specific processing methods of the risk review processing can be various. The following provides an optional processing method. As Figure 2 shown, it can specifically include the processing of the following steps S202 to S204.

[0087] In step S202, according to a pre-trained risk detection model, the target annotation result is processed for risk detection to determine the risk type corresponding to the target annotation result.

[0088] Among them, the risk identification model can be a model constructed based on a preset machine learning algorithm for identifying the risk type of the data to be detected. Specifically, for example, the risk identification model can be a model constructed based on a neural network algorithm for identifying the risk type of business data. Among them, the risk types can include high-risk, medium-risk, low-risk and other types.

[0089] In step S204, according to the risk type corresponding to the target annotation result, the risk review result for the data to be annotated is determined.

[0090] In implementation, the data to be annotated can be the data to be reviewed. The server can determine the risk review result for the data to be annotated according to the risk type corresponding to the target annotation result, so as to determine whether the data to be annotated is illegal data or risk data, etc. according to the risk review result.

[0091] An embodiment of this specification provides a data processing method. By receiving a labeling request for data to be labeled, the labeling request includes labeling requirement information corresponding to the data to be labeled, and the data to be labeled includes multimodal data. In response to the labeling request, a multimodal large language model is used to process the data to be labeled, and a first labeling result for the data to be labeled is determined. The labeling requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary labeling rules corresponding to each sub-requirement information, a data processing model corresponding to each sub-requirement information is determined. According to each data processing model, the information corresponding to the auxiliary labeling rules in the first labeling result is identified to obtain an auxiliary labeling result corresponding to each sub-requirement information. According to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result, a target labeling result for the data to be labeled is determined. In this way, in the case where the data to be labeled and the labeling requirements are relatively complex, on the one hand, through the understanding, summarization, and reasoning capabilities of the large language model, the first labeling result can be quickly and accurately determined. On the other hand, according to the data processing model determined by the auxiliary labeling rules corresponding to each sub-requirement information, the information corresponding to the auxiliary labeling rules in the first labeling result is identified to obtain the auxiliary labeling result. Furthermore, according to the first labeling result and the auxiliary labeling result that can meet the sub-requirement information, the target labeling result for the data to be labeled can be determined, improving the labeling efficiency and standard accuracy of data labeling.

[0092] An embodiment of this specification provides a data processing device. In the case where the data to be labeled and the labeling requirements are relatively complex, on the one hand, through the understanding, summarization, and reasoning capabilities of the large language model, the first labeling result can be quickly and accurately determined. On the other hand, according to the data processing model determined by the auxiliary labeling rules corresponding to each sub-requirement information, the information corresponding to the auxiliary labeling rules in the first labeling result is identified to obtain the auxiliary labeling result. Furthermore, according to the first labeling result and the auxiliary labeling result that can meet the sub-requirement information, the target labeling result for the data to be labeled can be determined, improving the labeling efficiency and standard accuracy of data labeling.

[0093] The above is the data processing method provided by the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a data processing device, as Figure 8 shown.

[0094] The data processing device includes: a request receiving module 801, a model processing module 802, a model determining module 803, an information identifying module 804, and a result determining module 805, where: The request receiving module 801 receives a labeling request for the data to be labeled, and the labeling request includes the labeling requirement information corresponding to the data to be labeled, and the data to be labeled includes multimodal data; The model processing module 802 processes the data to be labeled using a multimodal large language model in response to the labeling request, and determines a first labeling result for the data to be labeled; The model determination module 803 splits the labeling requirement information to obtain multiple sub-requirement information, and determines a data processing model corresponding to each sub-requirement information according to the auxiliary labeling rules corresponding to each sub-requirement information; The information recognition module 804 recognizes and processes the information corresponding to the auxiliary labeling rules in the first labeling result according to each data processing model, and obtains an auxiliary labeling result corresponding to each sub-requirement information; The result determination module 805 is used to determine a target labeling result for the data to be labeled according to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result.

[0095] In the embodiments of this specification, the model determination module screens out a data processing model corresponding to each sub-requirement information from a pre-constructed auxiliary tool library according to the auxiliary labeling rules corresponding to each sub-requirement information. The call protocols corresponding to the call interfaces of different data processing models in the auxiliary tool library are the same. The auxiliary tool library is a database obtained by uniformly integrating and encapsulating the auxiliary labeling capabilities, services, and interfaces involved in multimodal data and / or multiple business knowledge bases.

[0096] In the embodiments of this specification, the auxiliary labeling capabilities involved in the multimodal data in the auxiliary tool library include one or more of the algorithms for processing image data, text data, audio data, and video data.

[0097] In the embodiments of this specification, the algorithms for processing text data include one or more of a retrieval highlighting algorithm, a rewriting and correction algorithm, a summary and question answering algorithm, and a knowledge hint algorithm. The algorithms for processing image data include one or more of an optical character recognition algorithm, an element understanding algorithm, a graphic-text matching algorithm, and a summary and question answering algorithm. The algorithms for processing audio data include one or more of a speech recognition algorithm, an audio segmentation algorithm, an audio caption generation algorithm, and a voiceprint recognition algorithm. The algorithms for processing video data include one or more of a key frame recognition algorithm, a video understanding algorithm, a video segmentation algorithm, and a topic tracking algorithm.

[0098] In the embodiments of this specification, the service knowledge base includes one or more of an entity knowledge base, a question-and-answer knowledge base, an evaluation knowledge base, and a multi-modal knowledge base. Among them, the entity knowledge base includes one or more of risk entity knowledge data, financial product knowledge data, government and enterprise service knowledge data, and medical disease knowledge data. The question-and-answer knowledge base includes one or more of security risk knowledge data, insurance and financial management knowledge data, assistant question-and-answer knowledge data, and life service knowledge data. The evaluation knowledge base includes one or more of base capability knowledge data, security authentication knowledge data, financial compliance knowledge data, and medical and health knowledge data. The multi-modal knowledge base includes one or more of content security knowledge data, graphic and text understanding knowledge data, multi-modal voucher knowledge data, and voice intention knowledge data.

[0099] In the embodiments of this specification, the services integrated in the auxiliary tool library include one or more of the algorithm services, retrieval services, and knowledge management services involved in the auxiliary annotation capability.

[0100] The embodiments of this specification provide a data processing device. By receiving a annotation request for data to be annotated, the annotation request includes annotation requirement information corresponding to the data to be annotated, and the data to be annotated includes multi-modal data. In response to the annotation request, a multi-modal large language model is used to process the data to be annotated to determine a first annotation result for the data to be annotated. The annotation requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary annotation rules corresponding to each sub-requirement information, a data processing model corresponding to each sub-requirement information is determined. According to each data processing model, the information corresponding to the auxiliary annotation rules in the first annotation result is identified to obtain an auxiliary annotation result corresponding to each sub-requirement information. According to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result, a target annotation result for the data to be annotated is determined. In this way, in the case where the data to be annotated and the annotation requirements are relatively complex, on the one hand, through the understanding, summarization, and reasoning capabilities of the large language model, the first annotation result can be quickly and accurately determined. On the other hand, according to the data processing model determined by the auxiliary annotation rules corresponding to each sub-requirement information, the information corresponding to the auxiliary annotation rules in the first annotation result is identified to obtain the auxiliary annotation result. Furthermore, according to the first annotation result and the auxiliary annotation result that can meet the sub-requirement information, the target annotation result for the data to be annotated can be determined, improving the annotation efficiency and standard accuracy of data annotation.

[0101] The above is the data processing device provided by the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a data processing device, as Figure 9 shown.

[0102] The data processing device may be the terminal device or server provided in the above embodiments, etc.

[0103] Data processing devices can vary significantly due to differences in configuration or performance. They may include one or more processors 901 and a memory 902. The memory 902 can store one or more application programs or data. Among them, the memory 902 can be transient storage or persistent storage. The application programs stored in the memory 902 can include one or more modules (not shown in the figure), and each module can include a series of computer-executable instructions for the data processing device. Further, the processor 901 can be configured to communicate with the memory 902 and execute a series of computer-executable instructions in the memory 902 on the data processing device. The data processing device can also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input / output interfaces 905, and one or more keyboards 906.

[0104] Specifically, in this embodiment, the data processing device includes a memory and one or more programs. One or more of the programs are stored in the memory, and one or more of the programs can include one or more modules. Each module can include a series of computer-executable instructions for the data processing device and is configured to be executed by one or more processors. The one or more programs include the following computer-executable instructions: Receive a labeling request for the data to be labeled. The labeling request includes labeling requirement information corresponding to the data to be labeled, and the data to be labeled includes multimodal data; In response to the labeling request, use a multimodal large language model to process the data to be labeled and determine a first labeling result for the data to be labeled; Perform a splitting process on the labeling requirement information to obtain multiple sub-requirement information, and determine a data processing model corresponding to each sub-requirement information according to the auxiliary labeling rules corresponding to each sub-requirement information; According to each data processing model, perform an identification process on the information corresponding to the auxiliary labeling rules in the first labeling result to obtain an auxiliary labeling result corresponding to each sub-requirement information; Determine a target labeling result for the data to be labeled according to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result.

[0105] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiment of the data processing device, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the corresponding description in the method embodiment.

[0106] An embodiment of this specification provides a data processing device. By receiving a labeling request for data to be labeled, the labeling request includes labeling requirement information corresponding to the data to be labeled, and the data to be labeled includes multimodal data. In response to the labeling request, a multimodal large language model is used to process the data to be labeled, and a first labeling result for the data to be labeled is determined. The labeling requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary labeling rules corresponding to each sub-requirement information, a data processing model corresponding to each sub-requirement information is determined. According to each data processing model, the information corresponding to the auxiliary labeling rules in the first labeling result is identified to obtain an auxiliary labeling result corresponding to each sub-requirement information. According to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result, a target labeling result for the data to be labeled is determined. In this way, in the case where the data to be labeled and the labeling requirements are relatively complex, on the one hand, through the understanding, summarization, and reasoning capabilities of the large language model, the first labeling result can be quickly and accurately determined. On the other hand, according to the data processing model determined by the auxiliary labeling rules corresponding to each sub-requirement information, the information corresponding to the auxiliary labeling rules in the first labeling result is identified to obtain an auxiliary labeling result. Furthermore, according to the first labeling result and the auxiliary labeling result that can meet the sub-requirement information, the target labeling result for the data to be labeled can be determined, improving the labeling efficiency and standard accuracy of data labeling.

[0107] Further, based on the above Figures 1 to 7 shown method, one or more embodiments of this specification also provide a storage medium for storing computer-executable instruction information. In a specific embodiment, the storage medium can be a USB flash drive, an optical disc, a hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, the following process can be implemented: Receive a labeling request for data to be labeled, where the labeling request includes labeling requirement information corresponding to the data to be labeled, and the data to be labeled includes multimodal data; In response to the labeling request, use a multimodal large language model to process the data to be labeled and determine a first labeling result for the data to be labeled; The annotation requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary annotation rules corresponding to each sub-requirement information, a data processing model corresponding to each sub-requirement information is determined; According to each data processing model, the information corresponding to the auxiliary annotation rules in the first annotation result is identified to obtain an auxiliary annotation result corresponding to each sub-requirement information; According to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result, a target annotation result for the data to be annotated is determined.

[0108] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the above-mentioned storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0109] An embodiment of this specification provides a storage medium. By receiving an annotation request for data to be annotated, the annotation request includes annotation requirement information corresponding to the data to be annotated, and the data to be annotated includes multi-modal data. In response to the annotation request, a multi-modal large language model is used to process the data to be annotated to determine a first annotation result for the data to be annotated. The annotation requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary annotation rules corresponding to each sub-requirement information, a data processing model corresponding to each sub-requirement information is determined. According to each data processing model, the information corresponding to the auxiliary annotation rules in the first annotation result is identified to obtain an auxiliary annotation result corresponding to each sub-requirement information. According to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result, a target annotation result for the data to be annotated is determined. In this way, in the case where the data to be annotated and the annotation requirements are relatively complex, on the one hand, the understanding, summarization, and reasoning capabilities of the large language model can be used to quickly and accurately determine the first annotation result. On the other hand, the data processing model determined according to the auxiliary annotation rules corresponding to each sub-requirement information can be used to identify the information corresponding to the auxiliary annotation rules in the first annotation result to obtain the auxiliary annotation result. Furthermore, the target annotation result for the data to be annotated can be determined according to the first annotation result and the auxiliary annotation result that can meet the sub-requirement information, improving the annotation efficiency and standard accuracy of data annotation.

[0110] Further, based on the above Figures 1 to 7 shown method, one or more embodiments of this specification also provide a computer program product, including a computer program. When the computer program in this computer program product is executed by a processor, the following process can be implemented: Receive a labeling request for data to be labeled, where the labeling request includes labeling requirement information corresponding to the data to be labeled, and the data to be labeled includes multimodal data; In response to the labeling request, use a multimodal large language model to process the data to be labeled and determine a first labeling result for the data to be labeled; Perform a splitting process on the labeling requirement information to obtain multiple sub-requirement information, and determine a data processing model corresponding to each sub-requirement information according to the auxiliary labeling rule corresponding to each sub-requirement information; According to each data processing model, perform an identification process on the information corresponding to the auxiliary labeling rule in the first labeling result to obtain an auxiliary labeling result corresponding to each sub-requirement information; Determine a target labeling result for the data to be labeled according to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result.

[0111] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the above embodiment of a computer program product, since it is basically similar to the method embodiment, the description is relatively simple, and reference can be made to the partial description of the method embodiment for related parts.

[0112] An embodiment of this specification provides a computer program product. By receiving a labeling request for data to be labeled, the labeling request includes labeling requirement information corresponding to the data to be labeled, and the data to be labeled includes multimodal data. In response to the labeling request, a multimodal large language model is used to process the data to be labeled to determine a first labeling result for the data to be labeled. The labeling requirement information is split to obtain multiple sub-requirement information, and according to the auxiliary labeling rules corresponding to each sub-requirement information, a data processing model corresponding to each sub-requirement information is determined. According to each data processing model, the information corresponding to the auxiliary labeling rules in the first labeling result is identified to obtain an auxiliary labeling result corresponding to each sub-requirement information. According to the auxiliary labeling result corresponding to each sub-requirement information and the first labeling result, a target labeling result for the data to be labeled is determined. In this way, in the case where the data to be labeled and the labeling requirements are relatively complex, on the one hand, the first labeling result can be quickly and accurately determined through the understanding, summarization, and reasoning capabilities of the large language model. On the other hand, the information corresponding to the auxiliary labeling rules in the first labeling result can be identified according to the data processing model determined by the auxiliary labeling rules corresponding to each sub-requirement information to obtain the auxiliary labeling result. Furthermore, the target labeling result for the data to be labeled can be determined according to the first labeling result and the auxiliary labeling result that can meet the sub-requirement information, improving the labeling efficiency and standard accuracy of data labeling.

[0113] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0114] In the 1990s, it was obvious to distinguish whether an improvement to a technology was a hardware improvement (e.g., improvement to circuit structures such as diodes, transistors, switches, etc.) or a software improvement (improvement to method flows). However, with the development of technology, many improvements to method flows today can be regarded as direct improvements to hardware circuit structures. Almost all designers obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented with a hardware entity module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logical function is determined by a user's programming of the device. The designer can program by himself to "integrate" a digital system on a piece of PLD without asking a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating an integrated circuit chip, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a Hardware Description Language (HDL), and there is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow with the above-mentioned several hardware description languages and programming it into the integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0115] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0116] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0117] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0118] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0119] Embodiments of this specification are described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of this specification. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable serial-parallel devices for fraud cases to generate a machine, so that the instructions executed by the processor of the computer or other programmable serial-parallel devices for fraud cases generate a device for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks.

[0120] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable serial-parallel devices for fraud cases to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks.

[0121] These computer program instructions can also be loaded onto a computer or other programmable serial-parallel devices for fraud cases, so that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks.

[0122] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0123] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0124] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0125] It should also be noted that the term "comprises," "comprising," or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, commodity, or device that comprises a list of elements does not include only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, commodity, or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity, or device that comprises the element.

[0126] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0127] One or more embodiments of this specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0128] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiment.

[0129] The above is only the embodiment of this specification and is not intended to limit this document. For those skilled in the art, various changes and modifications can be made to this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this specification.

Claims

1. A data processing method, comprising: Receiving an annotation request for data to be annotated, where the annotation request includes annotation requirement information corresponding to the data to be annotated, and the data to be annotated includes multimodal data; In response to the annotation request, using a multimodal large language model to process the data to be annotated and determining a first annotation result for the data to be annotated; Performing a splitting process on the annotation requirement information to obtain multiple sub-requirement information, and determining a data processing model corresponding to each sub-requirement information according to the auxiliary annotation rule corresponding to each sub-requirement information; According to each data processing model, performing an identification process on the information in the first annotation result corresponding to the auxiliary annotation rule to obtain an auxiliary annotation result corresponding to each sub-requirement information; Determining a target annotation result for the data to be annotated according to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result.

2. The method according to claim 1, wherein the data processing model includes a duplicate identification model, a relevance identification model, and a knowledge assistance model. The duplicate identification model is used to identify duplicate information in the annotation result, the relevance identification model is used to identify information related to a preset keyword in the annotation result, and the knowledge assistance model is used to obtain knowledge related to the preset keyword in the annotation result.

3. The method according to claim 2, wherein determining the target annotation result for the data to be annotated according to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result includes: Obtaining a data rendering method corresponding to each auxiliary annotation rule; According to the data rendering method, respectively performing a rendering process on the auxiliary annotation result corresponding to each sub-requirement information, and determining the target annotation result for the data to be annotated according to the rendered auxiliary annotation result and the first annotation result.

4. The method according to claim 1, wherein determining the data processing model corresponding to each sub-requirement information according to the auxiliary annotation rule corresponding to each sub-requirement information includes: According to the auxiliary annotation rule corresponding to each sub-requirement information, screening out the data processing model corresponding to each sub-requirement information from a pre-constructed auxiliary tool library, and the call protocols corresponding to the call interfaces of different data processing models in the auxiliary tool library are the same.

5. The method according to claim 4, wherein the auxiliary tool library contains multiple data conversion models for processing multimodal data. The performing an identification process on the information in the first annotation result corresponding to the auxiliary annotation rule according to each data processing model to obtain an auxiliary annotation result corresponding to each sub-requirement information includes: Determining sub-annotation data in the data to be annotated corresponding to each sub-requirement information; Using the data conversion model in the auxiliary tool library to perform text conversion processing on the sub-annotation data to obtain the converted sub-annotation data; Based on each of the said data processing models, identify and process the information corresponding to the auxiliary annotation rules in the first annotation result based on the converted sub-annotation data and the first annotation result, to obtain the auxiliary annotation result corresponding to each of the said sub-requirement information.

6. The method according to claim 4, wherein determining the target annotation result for the data to be annotated based on the auxiliary annotation result corresponding to each of the said sub-requirement information and the first annotation result includes: Receiving an interactive retrieval request for a target keyword in the first annotation result; In response to the interactive retrieval request, determining the target knowledge corresponding to the target keyword; Based on the target knowledge, the auxiliary annotation result corresponding to each of the said sub-requirement information, and the first annotation result, determining the target annotation result for the data to be annotated.

7. The method according to claim 6, wherein determining the target knowledge corresponding to the target keyword includes: Performing intention recognition processing on the interactive retrieval request to obtain an intention recognition result; Using a large language model, determining the target knowledge corresponding to the target keyword based on the intention recognition result and the target keyword.

8. The method according to claim 6, wherein determining the target knowledge corresponding to the target keyword includes: According to the interactive retrieval request, screening out the target knowledge base corresponding to the interactive retrieval request from the pre-constructed auxiliary tool library; Based on the similarity between the target keyword and each knowledge stored in the target knowledge base, determining the target knowledge corresponding to the target keyword among the knowledge stored in the target knowledge base.

9. The method according to claim 1, the method further includes: Performing risk detection processing on the target annotation result according to a pre-trained risk detection model to determine the risk type corresponding to the target annotation result; Based on the risk type corresponding to the target annotation result, determining the risk review result for the data to be annotated.

10. A data processing device, comprising: A request receiving module, which receives an annotation request for data to be annotated, and the annotation request includes the annotation requirement information corresponding to the data to be annotated, and the data to be annotated includes multi-modal data; A model processing module, in response to the annotation request, uses a multi-modal large language model to process the data to be annotated to determine the first annotation result for the data to be annotated; A model determination module, which splits the annotation requirement information to obtain multiple sub-requirement information, and determines the data processing model corresponding to each sub-requirement information according to the auxiliary annotation rule corresponding to each sub-requirement information; An information recognition module, according to each of the data processing models, identifies and processes the information corresponding to the auxiliary annotation rules in the first annotation result to obtain the auxiliary annotation result corresponding to each sub-requirement information; A result determination module, based on the auxiliary annotation result corresponding to each of the said sub-requirement information and the first annotation result, determines the target annotation result for the data to be annotated.

11. The apparatus according to claim 10, wherein the model determination module screens out data processing models corresponding to each sub - requirement information from a pre - constructed auxiliary tool library according to the auxiliary annotation rules corresponding to each sub - requirement information. The call protocols corresponding to the call interfaces of different data processing models in the auxiliary tool library are the same. The auxiliary tool library is a database obtained by uniformly integrating and encapsulating the auxiliary annotation capabilities involved in multimodal data and / or multiple service knowledge bases, as well as services and interfaces.

12. The apparatus according to claim 11, wherein the auxiliary annotation capabilities involved in the multimodal data in the auxiliary tool library include one or more of the algorithms for processing image data, text data, audio data, and video data. The algorithms for processing text data include one or more of a retrieval highlighting algorithm, a rewriting and correction algorithm, a summary and question - answering algorithm, and a knowledge hint algorithm. The algorithms for processing image data include one or more of an optical character recognition algorithm, an element understanding algorithm, a graphic - text matching algorithm, and a summary and question - answering algorithm. The algorithms for processing audio data include one or more of a speech recognition algorithm, an audio segmentation algorithm, an audio caption generation algorithm, and a voiceprint recognition algorithm. The algorithms for processing video data include one or more of a key - frame recognition algorithm, a video understanding algorithm, a video segmentation algorithm, and a topic tracking algorithm.

13. The device according to claim 12, wherein the service knowledge base includes one or more of an entity knowledge base, a question and answer knowledge base, an evaluation knowledge base, and a multi-modal knowledge base, where The entity knowledge base includes one or more of risk entity knowledge data, financial product knowledge data, government - enterprise service knowledge data, and medical disease knowledge data. The question - answering knowledge base includes one or more of security risk knowledge data, insurance and wealth management knowledge data, assistant question - answering knowledge data, and life - service knowledge data. The evaluation knowledge base includes one or more of base - capacity knowledge data, security authentication knowledge data, financial compliance knowledge data, and medical and health knowledge data. The multimodal knowledge base includes one or more of content - security knowledge data, graphic - text understanding knowledge data, multimodal voucher knowledge data, and speech - intent knowledge data.

14. The apparatus according to claim 12, wherein the services integrated in the auxiliary tool library include one or more of the algorithm services, retrieval services, and knowledge management services involved in the auxiliary annotation capabilities.

15. A data - processing device, the data - processing device comprising: a processor; and a memory arranged to store computer - executable instructions, the executable instructions, when executed, causing the processor to: receive a annotation request for data to be annotated, the annotation request including annotation - requirement information corresponding to the data to be annotated, and the data to be annotated including multimodal data; in response to the annotation request, use a multimodal large language model to process the data to be annotated and determine a first annotation result for the data to be annotated; perform a splitting process on the annotation - requirement information to obtain a plurality of sub - requirement information, and determine data processing models corresponding to each sub - requirement information according to the auxiliary annotation rules corresponding to each sub - requirement information; According to each of the data processing models, identify and process the information corresponding to the auxiliary annotation rules in the first annotation result to obtain an auxiliary annotation result corresponding to each sub-requirement information; Determine the target annotation result for the data to be annotated according to the auxiliary annotation result corresponding to each sub-requirement information and the first annotation result.

Images