A data labeling system for a speed and slow system to work together

Abstract

The embodiment of the present application relates to a kind of data labeling system of fast and slow system cooperation, the system includes: data preprocessing module, slow system, fast system and labeling output module;Data preprocessing module is used to split image / chart content and text content to scientific literature file, and image sequence and literature text are sent to slow system, image sequence is sent to fast system;Slow system is used to carry out labeling suggestion reasoning according to image sequence and literature text and output high-dimensional image-text fusion feature to fast system in reasoning process;Fast system is used to carry out labeling task processing according to image sequence and the high-dimensional image-text fusion feature output by slow system;Labeling output module is used to output literature labeling report.Through the present application, it can guarantee labeling efficiency, and can improve labeling quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data labeling system for fast and slow system cooperative work. BACKGROUND

[0002] In the application field of artificial intelligence technology, document data labeling is a relatively common application scenario. At present, some lightweight small models (such as some customized machine learning models, deep learning models, etc.) are mostly used to process data labeling tasks in the form of entity triplets (entity text, entity type, entity confidence). The advantage of this small model processing mechanism is fast labeling speed, but the disadvantage is poor model generalization and insufficient labeling accuracy.

[0003] Large models (such as large language models, cross-modal / multi-modal large language models, etc.) have strong context understanding ability and Chain-of-Thought (COT) reasoning ability, and good model generalization. Based on large models to process labeling tasks, the advantage is that it can improve the labeling accuracy, but the disadvantage is that if only relying on large models for full-text labeling, the calculation amount is too large and the labeling speed is slow.

[0004] In order to improve the labeling quality while ensuring the labeling efficiency, we give a cooperative work framework based on the combination of large and small models. Under this cooperative work framework, on the one hand, the understanding ability of the large model can be used to enhance the identification features of the small model, and the small model can be used to complete the fast labeling, on the other hand, the Chain-of-Thought reasoning ability of the large model can also be used to give reasoning explanation to the labeling process while the small model is processing the labeling task. Based on this cooperative work framework, both labeling efficiency and labeling quality can be guaranteed. And the present application is the specific implementation scheme of this cooperative work framework. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art, provide a data labeling system with fast and slow system cooperation, which comprises a data preprocessing module, a slow system, a fast system and a labeling output module; wherein the data preprocessing module is connected with the slow system and the fast system respectively; the slow system is connected with the fast system; the labeling output module is connected with the slow system and the fast system respectively. The data preprocessing module is used for splitting the image / chart content and the text content of the input scientific literature file to obtain corresponding image sequences and literature text, and sending the image sequences to the slow system and the fast system; the slow system is used for labeling suggestion reasoning according to the image sequences and the literature text to obtain labeling suggestion instructions, and outputting high-dimensional image-text fusion features to the fast system during processing; the fast system is used for labeling task processing according to the image sequences and the high-dimensional image-text fusion features output by the slow system to obtain labeling entity sequences, and sending the labeling entity sequences to the labeling output module; and the labeling output module is used for forming corresponding literature labeling reports by combining the labeling suggestion instructions and the labeling object sequences, and outputting the literature labeling reports.

[0006] To achieve the above purpose, the embodiment of the present application provides a data labeling system with fast and slow system cooperation, which comprises a data preprocessing module, a slow system, a fast system and a labeling output module;

[0007] The data preprocessing module is connected with the slow system and the fast system respectively; the slow system is connected with the fast system; and the labeling output module is connected with the slow system and the fast system respectively.

[0008] The data preprocessing module is used for receiving an input scientific literature file; splitting the image / chart content and the text content of the scientific literature file to obtain corresponding first image sequences and first literature text; sending the first image sequences and the first literature text to the slow system; and sending the first image sequences to the fast system.

[0009] The slow system is used for labeling suggestion reasoning according to the first image sequences and the first literature text to obtain corresponding labeling suggestion instructions, and sending the labeling suggestion instructions to the labeling output module; and outputting high-dimensional image-text fusion features to the fast system during processing.

[0010] The fast system is used for labeling task processing according to the first image sequences and the high-dimensional image-text fusion features output by the slow system to obtain corresponding labeling entity sequences, and sending the labeling entity sequences to the labeling output module.

[0011] The labeling output module is used for forming corresponding literature labeling reports by combining the labeling suggestion instructions and the labeling object sequences, and outputting the literature labeling reports.

[0012] Preferably, the slow system comprises a visual encoder, a language encoder, a multi-modal feature fusion module and a thought chain reasoning module; the multi-modal feature fusion module is connected with the visual encoder, the language encoder and the thought chain reasoning module respectively; the multi-modal feature fusion module is also connected with the adapter module of the fast system; the visual encoder is realized based on a Vision Transformer model; the language encoder is realized based on a Decoder-only pre-training language model; and the thought chain reasoning module is realized based on a Transformer model.

[0013] The fast system is sequentially connected by a basic encoding module, the adapter module and a labeling decision module; the basic encoding module is realized based on a Vision Transformer model; and the labeling decision module is realized based on a Diffusion Transformer model.

[0014] Further, the language encoder at least has a 40-layer Transformer Decoder module.

[0015] The vocabulary size of the language encoder is not less than 151,936, and supports professional vocabulary expansion.

[0016] Preferably, when the slow system is used to send the corresponding labeling suggestion description obtained by the labeling suggestion reasoning according to the first image sequence and the first literature text to the labeling output module, the slow system is used to:

[0017] The visual encoder is used to receive the first image sequence sent by the data preprocessing module, and perform image feature encoding processing on the first image sequence to obtain a corresponding first visual feature vector, and send the first visual feature vector to the multi-modal feature fusion module;

[0018] The language encoder is used to receive the first literature text sent by the data preprocessing module, and perform text feature encoding processing on the first literature text to obtain a corresponding first text feature vector, and send the first text feature vector to the multi-modal feature fusion module;

[0019] The multi-modal feature fusion module is used to perform feature fusion processing on the first visual feature vector and the first text feature vector according to a cross-attention mechanism to obtain a corresponding first fusion feature vector, and send the first fusion feature vector to the thought chain reasoning module;

[0020] The thought chain reasoning module is used to perform multi-step reasoning according to the first fusion feature vector in a preset annotation suggestion chain reasoning paradigm, and send the corresponding annotation suggestion description to the annotation output module; the annotation suggestion description at least includes literature background description, keyword description, and annotation strategy description.

[0021] Preferably, the slow system is specifically used for sending the first fusion feature vector obtained by the multi-modal feature fusion module to the adapter module when outputting high-dimensional image-text fusion features to the fast system during processing.

[0022] Preferably, the fast system is specifically used for sending the corresponding annotation entity sequence obtained by performing annotation task processing according to the first image sequence and the high-dimensional image-text fusion features output by the slow system to the annotation output module.

[0023] The basic encoding module is used for receiving the first image sequence sent by the data preprocessing module, and performing image feature encoding processing on the first image sequence to obtain a corresponding second visual feature vector, which is sent to the adapter module.

[0024] The adapter module is used for regarding the feature vector space of the second visual feature vector as a low-dimensional space, regarding the feature vector space of the first fusion feature vector as a high-dimensional space, performing feature vector projection processing on the second visual feature vector from the low-dimensional space to the high-dimensional space to obtain a corresponding first projection feature vector, performing standardization processing on the first projection feature vector to obtain a corresponding first standardized vector, and performing feature fusion processing on the first standardized vector and the first fusion feature vector based on a skip connection manner to obtain a corresponding second fusion feature vector, which is sent to the annotation decision module.

[0025] The annotation decision module is used for performing annotation entity identification according to the second fusion feature vector to obtain a corresponding annotation entity sequence, and sending the annotation entity sequence to the annotation output module; the annotation entity sequence includes a plurality of annotation entities; each annotation entity is composed of an entity triple, and the entity triple includes entity text, entity type, and entity confidence.

[0026] The embodiment of the present application provides a data labeling system of fast-slow system cooperative work, which comprises a data preprocessing module, a slow system, a fast system and a labeling output module; wherein the data preprocessing module is connected with the slow system and the fast system respectively; the slow system is connected with the fast system; and the labeling output module is connected with the slow system and the fast system respectively. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A module structure diagram of the data labeling system of fast-slow system cooperative work provided by the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0029] As Figure 1 As shown in the module structure diagram of the data labeling system of fast-slow system cooperative work provided by the embodiment of the present application, the data labeling system provided by the embodiment of the present application mainly comprises a data preprocessing module 1, a slow system 2, a fast system 3 and a labeling output module 4. The data preprocessing module 1 is connected with the slow system 2 and the fast system 3 respectively; the slow system 2 is connected with the fast system 3; and the labeling output module 4 is connected with the slow system 2 and the fast system 3 respectively.

[0030] (I) Data preprocessing module 1:

[0031] The data preprocessing module 1 of the embodiment of the present application is used for receiving an input scientific literature file; and splitting the scientific literature file to obtain a corresponding first image sequence and a first literature text according to image / chart content and text content; and sending the first image sequence and the first literature text to the slow system 2; and sending the first image sequence to the fast system 3.

[0032] (ii) The slow system 2:

[0033] The slow system 2 of the embodiment of the present application comprises a visual encoder 21, a language encoder 22, a multi-modal feature fusion module 23 and a thinking chain reasoning module 24. The multi-modal feature fusion module 23 is connected with the visual encoder 21, the language encoder 22 and the thinking chain reasoning module 24 respectively; and the multi-modal feature fusion module 23 is also connected with the adapter module 32 of the fast system 3. The visual encoder 21 is realized based on a Vision Transformer model; the language encoder 22 is realized based on a Decoder-only pre-training language model; and the thinking chain reasoning module 24 is realized based on a Transformer model. In addition, the language encoder 22 of the embodiment of the present application at least has a 40-layer Transformer Decoder module; the vocabulary size of the language encoder 22 is not less than 151,936, and supports professional vocabulary expansion.

[0034] The slow system 2 is used for performing annotation suggestion reasoning according to the first image sequence and the first literature text to obtain a corresponding annotation suggestion description, and sending the annotation suggestion description to the annotation output module 4; and outputting high-dimensional image-text fusion features to the fast system 3 in the processing process.

[0035] It should be noted that the encoding and decoding performance of each module of the slow system 2 of the embodiment of the present application and the overall annotation suggestion reasoning function have been completed model training before application.

[0036] In a specific implementation manner of the embodiment of the present application, the slow system 2 is specifically used for, when sending the corresponding annotation suggestion description obtained by performing annotation suggestion reasoning according to the first image sequence and the first literature text to the annotation output module 4:

[0037] The visual encoder 21 is used for receiving the first image sequence sent by the data preprocessing module 1; and performing image feature coding processing on the first image sequence to obtain a corresponding first visual feature vector, and sending the first visual feature vector to the multi-modal feature fusion module 23;

[0038] The language encoder 22 is used for receiving the first literature text sent by the data preprocessing module 1; and performing text feature coding processing on the first literature text to obtain a corresponding first text feature vector, and sending the first text feature vector to the multi-modal feature fusion module 23;

[0039] The multimodal feature fusion module 23 is configured to perform feature fusion processing on the first visual feature vector and the first text feature vector in a cross-attention mechanism to obtain a corresponding first fusion feature vector, and send the first fusion feature vector to the thinking chain reasoning module 24.

[0040] The thinking chain reasoning module 24 is configured to perform multi-step reasoning on the first fusion feature vector according to a preset annotation suggestion chain reasoning paradigm to obtain a corresponding annotation suggestion explanation, and send the annotation suggestion explanation to the annotation output module 4. Here, the annotation suggestion explanation of the embodiment of the present application at least includes literature background explanation, keyword explanation, and annotation strategy explanation.

[0041] Here, the annotation suggestion chain reasoning paradigm of the embodiment of the present application is a pre-set chain reasoning paradigm based on multi-segment formatted prompt instructions, and this chain reasoning paradigm is used to perform reasoning on the annotation suggestion according to the annotation specification / template of the scientific literature data.

[0042] In another specific implementation manner of the embodiment of the present application, the slow system 2 is specifically configured to send the obtained first fusion feature vector to the adapter module 32 by the multimodal feature fusion module 23 when outputting the high-dimensional image-text fusion feature to the fast system 3 in the processing process. Here, the multimodal feature fusion module 23 plays a role of feature transmission by sending the first fusion feature vector to the adapter module 32, and through this feature transmission, the modal understanding ability of the fast system 3 can be quickly improved without increasing the complexity of the fast system 3, so as to achieve the purpose of improving the annotation precision / quality.

[0043] (Three) fast system 3:

[0044] The fast system 3 of the embodiment of the present application is sequentially connected by the basic encoding module 31, the adapter module 32, and the annotation decision module 33; wherein the basic encoding module 31 is realized based on the Vision Transformer model; and the annotation decision module 33 is realized based on the Diffusion Transformer model.

[0045] The fast system 3 is configured to perform annotation task processing according to the first image sequence and the high-dimensional image-text fusion feature output by the slow system 2 to obtain a corresponding annotation entity sequence, and send the annotation entity sequence to the annotation output module 4.

[0046] It should be noted that the coding and decoding performance of each module of the fast system 3 of the embodiment of the present application and the overall annotation task processing function have been completed model training before application.

[0047] In another specific implementation manner of the embodiment of the present application, the fast system 3 is specifically configured to perform annotation task processing according to the first image sequence and the high-dimensional image-text fusion feature output by the slow system 2 to obtain a corresponding annotation entity sequence, and send the annotation entity sequence to the annotation output module 4.

[0048] The basic coding module 31 is configured to receive the first image sequence sent by the data preprocessing module 1, and perform image feature coding processing on the first image sequence to obtain a corresponding second visual feature vector.

[0049] The adapter module 32 is configured to regard the feature vector space of the second visual feature vector as a low-dimensional space, regard the feature vector space of the first fusion feature vector as a high-dimensional space, and perform feature vector projection processing on the second visual feature vector from the low-dimensional space to the high-dimensional space to obtain a corresponding first projected feature vector.

[0050] The labeling decision module 33 is configured to perform labeling entity recognition on the second fusion feature vector to obtain a corresponding labeling entity sequence, and send the labeling entity sequence to the labeling output module 4.

[0051] (Four) The labeling output module 4:

[0052] The labeling output module 4 of the embodiment of the application is configured to form a corresponding literature labeling report by combining the labeling suggestion description and the labeling object sequence, and output the literature labeling report.

[0053] It should be noted that the division of each module of the above system is only a logical functional division, and all or part of the modules can be integrated into one physical entity, or can be physically separated. These modules can all be implemented in the form of software called by a processing element; they can all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the adapter module can be a separately established processing element, or can be integrated in a chip of the above device, in addition, it can also be stored in the form of program code in the memory of the above device, and the functions of the above determination module can be called and executed by a processing element of the above system. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or can be independently implemented. The processing element described herein can be an integrated circuit having a signal processing capability. In the implementation process, each method step of the foregoing method or each module processing step of the foregoing system can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0054] For example, these modules in the above system can be one or more integrated circuits configured to implement the aforementioned methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module in the above system is implemented through processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0055] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0056] The embodiment of the present application provides a data labeling system of fast and slow system cooperative work, which comprises a data preprocessing module, a slow system, a fast system and a labeling output module; wherein the data preprocessing module is connected with the slow system and the fast system respectively; the slow system is connected with the fast system; and the labeling output module is connected with the slow system and the fast system respectively. The data preprocessing module is used for splitting the input scientific literature file to obtain corresponding image sequence and literature text content, and sending the image sequence to the slow system and the fast system; the slow system is used for performing labeling suggestion reasoning according to the image sequence and the literature text content to obtain labeling suggestion instructions, and outputting high-dimensional image-text fusion features to the fast system in the processing process; the fast system is used for performing labeling task processing according to the image sequence and the high-dimensional image-text fusion features output by the slow system to obtain labeling entity sequence, and sending the labeling entity sequence to the labeling output module; and the labeling output module is used for composing the labeling suggestion instructions and the labeling object sequence to obtain corresponding literature labeling report and outputting the literature labeling report. Through the fast and slow system cooperative work framework of the embodiment of the present application, the labeling efficiency is ensured, and the labeling quality is improved.

[0057] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, various aspects of each example have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation should not be interpreted to change the scope of the present application.

[0058] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0059] The above detailed description sets forth the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data annotation system for coordinated operation of fast and slow systems, characterized in that, The system includes: a data preprocessing module, a slow system, a fast system, and a labeling output module; The data preprocessing module is connected to both the slow system and the fast system; the slow system is connected to the fast system; the annotation output module is connected to both the slow system and the fast system. The data preprocessing module is used to receive input scientific literature files; and to split the scientific literature files into image / figure content and text content to obtain corresponding first image sequences and first literature text; and to send the first image sequences and first literature text to the slow system; and to send the first image sequences to the fast system; The slow system is used to perform annotation suggestion reasoning based on the first image sequence and the first document text to obtain corresponding annotation suggestion descriptions and send them to the annotation output module; and during the processing, it outputs high-dimensional image-text fusion features to the fast system; The fast system is used to perform annotation task processing based on the first image sequence and the high-dimensional image-text fusion features output by the slow system to obtain the corresponding annotation entity sequence and send it to the annotation output module. The annotation output module is used to combine the annotation suggestion description and the annotation entity sequence into a corresponding literature annotation report and output it. The slow system includes a visual encoder, a language encoder, a multimodal feature fusion module, and a thought chain reasoning module; the multimodal feature fusion module is connected to the visual encoder, the language encoder, and the thought chain reasoning module respectively; the multimodal feature fusion module is also connected to the adapter module of the fast system; the fast system is composed of a basic encoding module, the adapter module, and an annotation decision module connected sequentially. The visual encoder and the language encoder of the slow system respectively encode the first image sequence and the first document text using corresponding image and text feature encoding to obtain corresponding first visual feature vectors and first text feature vectors. The multimodal feature fusion module performs feature fusion on the first visual feature vector and the first text feature vector to obtain corresponding first fused feature vectors, and sends the first fused feature vector as the high-dimensional image-text fusion feature output by the slow system to the adapter module of the fast system.

2. The data annotation system for coordinated operation of fast and slow systems according to claim 1, characterized in that, The visual encoder is implemented based on the Vision Transformer model; the language encoder is implemented based on a decoder-only pre-trained language model; and the thought chain reasoning module is implemented based on the Transformer model. The basic encoding module is implemented based on the Vision Transformer model; the annotation decision module is implemented based on the Diffusion Transformer model.

3. The data annotation system for coordinated operation of fast and slow systems according to claim 2, characterized in that, The language encoder has at least a 40-layer Transformer Decoder module; The language encoder has a vocabulary size of no less than 151,936 words and supports the expansion of specialized vocabulary.

4. The data annotation system for coordinated operation of fast and slow systems according to claim 2, characterized in that, The slow system is specifically used when the corresponding annotation suggestion description is obtained by inferring annotation suggestions based on the first image sequence and the first document text and sent to the annotation output module: The visual encoder is used to receive the first image sequence sent by the data preprocessing module; and to perform image feature encoding processing on the first image sequence to obtain the corresponding first visual feature vector, which is then sent to the multimodal feature fusion module. The language encoder is used to receive the first document text sent by the data preprocessing module; and to perform text feature encoding processing on the first document text to obtain the corresponding first text feature vector, which is then sent to the multimodal feature fusion module. The multimodal feature fusion module is used to perform feature fusion processing on the first visual feature vector and the first text feature vector according to the cross-attention mechanism to obtain the corresponding first fused feature vector and send it to the thinking chain reasoning module; The thought chain reasoning module is used to perform multi-step reasoning based on the first fused feature vector according to a preset annotation suggestion chain reasoning paradigm to obtain the corresponding annotation suggestion description and send it to the annotation output module. The annotation suggestions should include at least a description of the literature background, keywords, and annotation strategies.

5. The data annotation system for coordinated operation of fast and slow systems according to claim 2, characterized in that, Specifically, when the slow system outputs high-dimensional image-text fusion features to the fast system during processing, the multimodal feature fusion module sends the obtained first fusion feature vector to the adapter module.

6. The data annotation system for coordinated operation of fast and slow systems according to claim 2, characterized in that, The fast system is specifically used when the corresponding labeled entity sequence is obtained by the labeling task processing based on the first image sequence and the high-dimensional image-text fusion features output by the slow system, and then sent to the labeling output module: The basic encoding module is used to receive the first image sequence sent by the data preprocessing module; and to perform image feature encoding processing on the first image sequence to obtain the corresponding second visual feature vector and send it to the adapter module; The adapter module is used to treat the feature vector space of the second visual feature vector as a low-dimensional space and the feature vector space of the first fused feature vector as a high-dimensional space, and to perform feature vector projection processing from the low-dimensional space to the high-dimensional space on the second visual feature vector to obtain the corresponding first projected feature vector; and to perform standardization processing on the first projected feature vector to obtain the corresponding first standardized vector; and to perform feature fusion processing on the first standardized vector and the first fused feature vector based on a skip connection method to obtain the corresponding second fused feature vector, which is then sent to the annotation decision module. The annotation decision module is used to identify the corresponding annotated entity sequence based on the second fusion feature vector; and send the annotated entity sequence to the annotation output module; the annotated entity sequence includes multiple annotated entities; each annotated entity is composed of an entity triple, the entity triple including entity text, entity type, and entity confidence.

Images