Interactive training data real-time collecting and labeling system and method for intelligent training

Abstract

The invention relates to the technical field of data annotation, in particular to an interactive training data real-time acquisition and annotation system and method for intelligent performance, and the system comprises a sensor assembly, a data acquisition module, an interaction module and a main control module, and the data acquisition module comprises a camera and a microphone; an audio analysis module, an image analysis module and a comprehensive mark processing center are arranged in the main control module; the main control module is connected with the sensor assembly, the data acquisition module and the interaction module. An image is shot, an operator adopts a gesture to indicate a target object, information of the target object is dictated, and sensor parameters are obtained through a sensor assembly; framing a target object in the image through a target frame, and converting the audio information into text information; and superimposing the character information, the sensor parameters and the target frame into the image, and labeling the target object. The target object is captured in real time and automatically annotated while the image is acquired each time, so that the annotated data is actively and quickly acquired.

Description

Technical Field

[0001] The present application relates to the field of data annotation technology, and in particular to a real-time collection and annotation system and method for interactive training data for embodied intelligence. Background Art

[0002] Currently, artificial intelligence has penetrated into all walks of life. Artificial intelligence is mainly a special large model driven by data, and the performance and ability of the large model mainly depend on the scale of the model structure or model parameters and the scale and quality of the data; it can be said that the ability and performance of the large model depends to a large extent on the data set for training the large model, and the original data set often contains a lot of noise information, error information, useless information, and redundant information, which requires manual data collection, cleaning, and labeling. The current large model training paradigm is: first, the original data is manually labeled with a labeling tool, and the labeled data can be used to train high-quality and high-performance large models. However, according to the current public Open data shows that the word metadata of mainstream 70B-level large model training data is generally higher than 1-2TB, and with the deepening of research, larger models at the trillion level and more data are the current research trend. However, the currently known public data sets are close to exhaustion, and the amount of labeled data used for training is close to exhaustion. Many research institutions have also studied various methods to obtain more high-quality labeled data. However, considering the scale of the current large models, manually labeling massive amounts of original training data is not only a huge workload, but also requires a lot of time, so the cost is extremely high, which is also a major obstacle and pain point that currently restricts the further development of large models.

[0003] In addition, the current training data collection and labeling process is based on first collecting and then manually labeling, while the data collection speed is very fast and the cost is relatively low, and the bottleneck is mainly the labeling work. In order to solve this problem, the present invention proposes a large model training data real-time collection and interactive labeling system to realize interactive and intelligent labeling while collecting training data, greatly improving the efficiency of data collection and labeling, and this new paradigm plays an important role in the rapid development of large models in the future. Summary of the invention

[0004] The present invention is intended to solve one of the problems existing in the background technology.

[0005] To this end, the present invention provides a real-time collection and annotation system and method for interactive training data for embodied intelligence.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A real-time collection and annotation system for interactive training data for embodied intelligence, comprising:

[0008] A sensor component, wherein the sensor component is used to sense environmental information;

[0009] A data acquisition module, wherein the data acquisition module includes a camera and a microphone, wherein the camera and the microphone are used to collect image information and audio information respectively;

[0010] A main control module, wherein the main control module is provided with an audio analysis module, an image analysis module, a comprehensive marking processing center and a database, and the main control module is connected to the sensor component and the data acquisition module;

[0011] An interactive module is connected to the main control module and is used to transmit information to the operator to realize information feedback and confirmation.

[0012] Furthermore, the data acquisition module also includes a speaker, and the microphone, the speaker and the main control module are connected to form a two-way information transmission.

[0013] Furthermore, a gesture recognition module is provided in the image analysis module to realize action recognition.

[0014] A method for real-time collection and annotation of interactive training data for embodied intelligence, which is implemented based on the above-mentioned real-time collection and annotation system for interactive training data for embodied intelligence, is characterized by comprising the following steps:

[0015] Data acquisition: The camera captures the image, and the operator indicates the target object with gestures or fingers, and verbally states the information of the target object, and obtains the sensor parameters through the sensor component;

[0016] Data analysis: Frame the target object indicated by the gesture in the image with a target box and convert the audio information into text information;

[0017] Comprehensive calibration: superimpose the acquired text information, sensor parameters and target frame into the image to mark the target object;

[0018] Review the marking results and confirm the marking results again.

[0019] Furthermore, in the data analysis step, the image data analysis is to perform target recognition on the image acquired by the camera, and the process specifically includes:

[0020] Pre-store gesture action instruction types and gesture matrix data in the gesture recognition module;

[0021] Determine whether the gesture in the image is the indicated gesture in the action instruction. If so, calculate the fingertip coordinates and output them to the gesture recognition matrix to proceed to the next step;

[0022] Perform preliminary target recognition on the captured image and frame all objects in the image using a target frame;

[0023] The vector distance is calculated based on the center coordinates of the general target frame and the coordinates of the fingertip of the indicating gesture; all target frames are traversed, the vector distance is calculated, and the target with the minimum distance is taken as the target associated with the indicating gesture, so as to select the target object, record the target object and store it.

[0024] Furthermore, in the gesture recognition module, a preset gesture sequence image is first obtained, and then the gesture is recognized through a hand detection and segmentation recognition algorithm, the gesture is modeled and a gesture frame is generated, and finally the gesture is modeled and a gesture frame is generated to recognize the gesture.

[0025] Furthermore, in the data analysis step, the microphone collects the oral content to form audio information and transmits it to the audio analysis module, converts it into text information, and identifies keywords in the text information to obtain the annotation information of the target object, and the annotation information includes color attributes, material attributes, size attributes, time information, and location information.

[0026] Furthermore, in the audio analysis module, a long audio information segment is segmented according to the sound pause interval, and then the segmented multiple audio information segments are converted into text information, and keywords are captured from the text information.

[0027] Furthermore, by comparing the text information lengths of the text information segments where the multiple annotation information are located, the multiple annotation information are sorted in order of importance, and the annotation information with longer text information length is arranged in front, and the target object is annotated.

[0028] Furthermore, in the review step, the annotation results are fed back to the operator through the display for review and confirmation. In this step, the operator provides feedback through a confirmation gesture or a cancellation gesture, or an interactive module. After confirming the annotation results, the results are saved in the database to form a new annotation data.

[0029] The beneficial effect of the present invention is that the present application can capture the target object in real time and automatically annotate it at the same time as each image is acquired, thereby realizing active and rapid acquisition of annotation data to enrich the annotation database. The present application adopts an interactive data acquisition method to greatly improve the efficiency of data acquisition and annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0031] Figure 1 It is a structural diagram of the real-time collection and annotation system of interactive training data for embodied intelligence in the present invention.

[0032] Figure 2 It is an implementation principle diagram of the real-time collection and annotation system of interactive training data for embodied intelligence in the present invention.

[0033] Figure 3 It is a flow chart of the real-time collection and labeling method of interactive training data for embodied intelligence in the present invention.

[0034] Figure 4 It is a schematic diagram of the gesture instruction in the present invention.

[0035] Figure 5 It is a schematic diagram of the gesture recognition method in the present invention.

[0036] Figure 6 It is a schematic diagram of the comprehensive labeling process in the present invention. DETAILED DESCRIPTION

[0037] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0038] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0039] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0040] A real-time collection and annotation system for interactive training data for embodied intelligence, referring to Figure 1, including sensor components, data acquisition modules, main control modules, interactive modules and power systems. The power system supplies power to the sensor components, data acquisition modules, main control modules, interactive modules and power systems. The sensor components, data acquisition modules and interactive modules all transmit information with the main control module through the communication module.

[0041] Specifically, the sensor assembly includes a humidity sensor, a temperature sensor, an electronic gyroscope, an acceleration sensor, a GPS, and other sensing devices for sensing environmental information. The data acquisition module includes a camera, a microphone, and a speaker. Multiple data acquisition modules can be provided, wherein the microphone, the speaker, and the main control module are connected to form a two-way information transmission, thereby realizing information interaction. For example, the microphone can receive sound signals, and the main control module can feedback information to the operator through the speaker.

[0042] The interactive module may include a display, interactive buttons, and a touch screen. The main control module may display the acquired and marked information through the display, and the operator may provide information feedback and confirmation through the interactive buttons and the touch screen.

[0043] The main control module integrates an audio analysis module, an image analysis module, a comprehensive marking processing center (CMPC) and a database. The audio analysis module is connected to the microphone to receive audio data and convert it into text information; the image analysis module is connected to the camera to receive image information and identify objects therein; the comprehensive marking processing center is used to integrate and calibrate the target object and annotate the identified text information to form marking data. It should be noted that a gesture recognition module is provided in the image analysis module to realize action recognition, so that the camera can recognize the operator's actions to judge the operator's instructions to realize non-contact interaction between the operator and the system. Specifically, the operator can use the following methods: Figure 4 The gestures shown are used to interact, where the start labeling task gesture indicates the start of the labeling job, the indication gesture is used to indicate the target object, the confirmation gesture is used to indicate information confirmation, the cancel gesture indicates that the information is wrong, and the end labeling task gesture is used to indicate that the labeling is completed.

[0044] A real-time collection and annotation method for interactive training data for embodied intelligence, comprising the following steps:

[0045] Step 1: Data Acquisition

[0046] Start the device, initialize the device, and start the sensor component and data acquisition module after the system self-checks. The sensor component and data acquisition module can work normally and start subsequent operations. At this time, the operator can start the labeling task by making a gesture to the camera, or by voice command, or by buttons or touch screen.

[0047] The sensor component parameters when shooting images are obtained in real time. The image is shot by the camera. At this time, the operator can use gestures or fingers to indicate the target object and verbally describe the information of the target object. The microphone collects the oral content to form audio information.

[0048] Step 2: Data Analysis

[0049] S2.1 Image data analysis: target recognition of images acquired by the camera:

[0050] S2.1.1 Pre-store gesture action instruction types and gesture matrix data in the gesture recognition module. Specifically, first obtain Figure 4 The gesture sequence images shown are then used to identify the gestures through hand detection and segmentation recognition algorithms, model the gestures and generate gesture frames, and finally model the gestures and generate gesture frames;

[0051] S2.1.2 Determine whether the gesture in the image is the indicating gesture in the action command. If so, calculate the fingertip coordinates and output them to the gesture recognition matrix to proceed to the next step. If not, report an error, and inform the operator to change the indicating gesture and retake the image through the interactive module in the form of display error or voice error.

[0052] S2.1.3 Perform preliminary target recognition on the captured image and frame all objects in the image using a target frame;

[0053] S2.1.4 Calculate the vector distance based on the center coordinates of the general target frame and the coordinates of the fingertips of the pointing gesture;

[0054] S2.1.5 traverses all target frames, calculates vector distances, takes the target with the minimum distance as the target associated with the indicating gesture, thereby selecting the target object, recording the target object and storing it.

[0055] S2.2 Audio Data Analysis

[0056] The microphone collects the spoken content to form audio information and transmits it to the audio analysis module, which converts it into text information and transmits it to the CMPC. The length of the text information is not limited, and it can be a brief description of the attributes of the item or a detailed description of the item from multiple angles.

[0057] In this step, the attributes of the item are described; the voice may describe from multiple angles, so that the text is parsed and converted into labeled data.

[0058] Specifically, a long audio message is segmented by the sound pause interval, with a sound pause of 3-5 seconds as a segmentation point;

[0059] Then, the segmented audio information is converted into text information, and the text information is captured by keywords, such as identifying color attributes, material attributes, size attributes, time information, location information, etc., so as to obtain multiple annotation information of the target object and send it to CMPC;

[0060] Furthermore, the multiple annotation information may be ranked in importance by comparing the text information lengths of the text information segments where the multiple annotation information are located, and the annotation information with longer text information lengths may be arranged in front, so as to annotate the target object.

[0061] The S2.3 system aggregates all raw data of images, audio, and sensor parameters for digital metadata collection, which is then sent to the database for storage and the raw data is sent to the CMPC as a reference set.

[0062] Step 3: Comprehensive Calibration

[0063] In the comprehensive labeling processing center CMPC, the acquired text annotation information, reference set and target frame of minimum vector distance are superimposed on the image to label the target object.

[0064] Step 4: Review the calibration results

[0065] After labeling, the labeling image formed by CMPC is fed back to the operator through the display for review and confirmation. Furthermore, the speaker can also broadcast the labeling information for the operator to confirm. In this step, the operator can give feedback through confirmation gestures, cancel gestures, and end the labeling task gesture order labeling operation. After confirming the labeling result, the result is saved to the database to form a new labeling data.

[0066] Furthermore, the trained general large model can be used to make review decisions. When the decision probability is low, for example, less than 50%, it can be handed over to manual judgment, thereby greatly improving the labeling efficiency.

[0067] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A real-time collection and annotation system for interactive training data for embodied intelligence, characterized in that: include, A sensor component, wherein the sensor component is used to sense environmental information; A data acquisition module, wherein the data acquisition module includes a camera and a microphone, wherein the camera and the microphone are used to collect image information and audio information respectively; A main control module, wherein the main control module is provided with an audio analysis module, an image analysis module, a comprehensive marking processing center and a database, and the main control module is connected to the sensor component and the data acquisition module; An interactive module is connected to the main control module and is used to transmit information to the operator to realize information feedback and confirmation.

2. The interactive training data real-time collection and annotation system for embodied intelligence according to claim 1, characterized in that: The data acquisition module also includes a speaker, and the microphone, the speaker and the main control module are connected to form a two-way information transmission.

3. The interactive training data real-time collection and annotation system for embodied intelligence according to claim 1, characterized in that: The image analysis module is provided with a gesture recognition module to realize action recognition.

4. A method for real-time collection and annotation of interactive training data for embodied intelligence, the method is implemented based on the real-time collection and annotation system for interactive training data for embodied intelligence as claimed in claim 3, characterized in that: The following steps are included: Data acquisition: The camera captures the image, and the operator indicates the target object with gestures or fingers, and verbally states the information of the target object, and obtains the sensor parameters through the sensor component; Data analysis: Frame the target object indicated by the gesture in the image with a target box and convert the audio information into text information; Comprehensive calibration: superimpose the acquired text information, sensor parameters and target frame into the image to mark the target object; Review the marking results and confirm the marking results again.

5. The real-time collection and annotation method for interactive training data for embodied intelligence according to claim 4, characterized in that: In the data analysis step, the image data analysis is to perform target recognition on the image acquired by the camera, and the process specifically includes: Pre-store gesture action instruction types and gesture matrix data in the gesture recognition module; Determine whether the gesture in the image is the indicated gesture in the action instruction. If so, calculate the fingertip coordinates and output them to the gesture recognition matrix to proceed to the next step; Perform preliminary target recognition on the captured image and frame all objects in the image using a target frame; The vector distance is calculated based on the center coordinates of the general target frame and the coordinates of the fingertip of the indicating gesture; all target frames are traversed, the vector distance is calculated, and the target with the minimum distance is taken as the target associated with the indicating gesture, so as to select the target object, record the target object and store it.

6. The real-time collection and annotation method for interactive training data for embodied intelligence according to claim 5, characterized in that: In the gesture recognition module, a preset gesture sequence image is first obtained, and then the gesture is recognized through the hand detection and segmentation recognition algorithm, the gesture is modeled and a gesture frame is generated, and finally the gesture is modeled and a gesture frame is generated to recognize the gesture.

7. The real-time collection and annotation method for interactive training data for embodied intelligence according to claim 4, characterized in that: In the data analysis step, the microphone collects the oral content to form audio information and transmits it to the audio analysis module, which converts it into text information and identifies keywords in the text information to obtain the annotation information of the target object, which includes color attributes, material attributes, size attributes, time information, and location information.

8. The method for real-time collection and annotation of interactive training data for embodied intelligence according to claim 7, characterized in that: In the audio analysis module, a long audio message is segmented according to the sound pause intervals, and then the segmented multiple audio messages are converted into text messages, and keywords are captured from the text messages.

9. The method for real-time collection and annotation of interactive training data for embodied intelligence according to claim 8, characterized in that: By comparing the text information lengths of the text information segments where the multiple annotation information are located, the multiple annotation information are sorted in order of importance, and the annotation information with longer text information length is arranged in front, and the target object is annotated.

10. The method for real-time collection and annotation of interactive training data for embodied intelligence according to claim 4, characterized in that: In the review step, the annotation result is fed back to the operator through the display for review and confirmation. In this step, the operator provides feedback through a confirmation gesture or a cancellation gesture, or an interactive module. After confirming the annotation result, the result is saved in the database to form a new annotation data.

Images