Artificial intelligence-based data collection robot corpus collection and annotation method

Abstract

The application discloses a data collection robot corpus collection and labeling method based on artificial intelligence, and comprises the following steps: collecting robot corpus original data to obtain an original data set; performing data standardization processing to obtain standardized data; performing multi-source behavior time sequence mapping processing to obtain a continuous behavior representation sequence; performing FLUSS semantic working condition segmentation processing to determine semantic working condition switching boundaries; extending a preset range forward and backward to each semantic working condition switching boundary to generate a key action interval; performing interval labeling processing to obtain a pre-labeling result; performing artificial review and correction on the pre-labeling result to obtain labeled data, and simultaneously performing behavior model training, connecting the trained behavior model to a real robot for testing, and outputting robot corpus data. The application improves the automation degree, key action recognition accuracy and corpus construction efficiency of robot corpus collection and labeling.

Description

Technical Field

[0001] This invention relates to the field of multimodal data annotation technology, and in particular to a method for collecting and annotating corpora of data acquisition robots based on artificial intelligence. Background Technology

[0002] With the continuous development of embodied intelligence, service robots, and industrial robot technologies, the application of robots in scenarios such as grasping, handling, inspection, interaction, and collaborative operation is constantly expanding. During the execution of tasks, robots continuously generate a large amount of data related to the task process, which typically includes video data, state field data, behavior control data, behavior data, and source data from different hardware interfaces and sensor interfaces. This data is not only an important foundation for robot behavior understanding, action learning, and model training, but also a key resource for building robot corpus data, supporting subsequent annotation training, and real robot verification.

[0003] In existing technologies, robot corpus collection and annotation mostly rely on decentralized collection and manual processing. The data structures generated by different robot manufacturers' platforms, common components, and external sensors often vary significantly, with inconsistent data formats, inconsistent time dimensions, and incomplete correspondence of field meanings. This often requires extensive manual cleaning and transformation of the raw data before it enters the annotation and training stages. At the same time, robots generate continuous behavioral process data when performing a single task. Existing technologies usually lack a temporal organization method for the task process, making it difficult to accurately identify continuous stage changes related to task behavior from multi-source data. This makes it difficult to stably determine subsequent annotation objects and results in low corpus processing efficiency.

[0004] Therefore, how to provide a data collection and annotation method for AI-based data acquisition robots is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method for collecting and annotating robot corpora based on artificial intelligence. This invention improves the automation level, accuracy of key action recognition, and efficiency of corpus construction by standardizing the original robot corpus data, mapping multi-source behavior time sequence, segmenting FLUSS semantic working conditions, generating key action intervals, and performing interval-based automatic pre-annotation.

[0006] The data acquisition and annotation method for AI-based data acquisition robots according to embodiments of the present invention includes the following steps:

[0007] Collect raw data from the robot's speech corpus to obtain the raw dataset;

[0008] Perform data standardization on the original dataset to obtain standardized data;

[0009] Perform multi-source behavioral temporal mapping processing on standardized data, establish task units and task phase units, and perform feature extraction and temporal mapping to obtain a continuous behavioral representation sequence;

[0010] FLUSS semantic condition segmentation is performed on the continuous behavior representation sequence. Arc connection relationship is constructed based on the nearest neighbor relationship of the subsequence of the continuous behavior representation sequence. The corresponding arc correction curve is calculated, and the semantic condition switching boundary is determined based on the local trough position of the arc correction curve.

[0011] The preset range is extended forward and backward for each semantic working condition switching boundary to form a boundary association window, and key action ranges are generated based on the boundary association window relationship;

[0012] Based on the key action intervals, interval labeling processing is performed. Keyframes are extracted and key node segments are generated within the data segments corresponding to each key action interval. Automatic pre-labeling is then performed to obtain the pre-labeling results.

[0013] The pre-labeled results are manually reviewed and corrected to obtain labeled data. At the same time, the behavior model is trained, the working condition fingerprint is constructed, and the collected results are supplemented and key reviews are performed. The trained behavior model is connected to a real robot for testing, and the robot's corpus data is output.

[0014] Optionally, obtaining the original dataset specifically includes:

[0015] By connecting to robot manufacturer platform interfaces, general component underlying call interfaces, and self-developed sensor interfaces, the robot calls the data acquisition channels of different cameras on the robot body to collect video data, status field data, behavior control data, and behavior data during the execution of a single task, forming initial data collection.

[0016] The initial collected data is processed for source and time correspondence, and the initial collected data is mapped to the collection interface and collection channel that generated the data. The data is also arranged in order according to the collection time to form associated collected data.

[0017] Read the task start and end information corresponding to the associated collected data, and perform task-level aggregation of the associated collected data according to the execution process of a single robot task. Combine the data within the same task start and end range into a data set of the same task to obtain the original dataset.

[0018] Optionally, obtaining the standardized data specifically includes:

[0019] Read the original dataset and classify and organize it according to the data source and task correspondence to form data to be standardized;

[0020] To standardize data, field uniform processing is performed, converting corresponding fields from different data sources into fields with uniform names, uniform order, and uniform types, thus forming field-standardized data.

[0021] The time-standardized data of the fields is uniformly processed by aligning the time according to the collection time of data from various sources within the same task to form time-standardized data.

[0022] Anomaly handling is performed on time-standardized data to identify missing, duplicate, conflicting, and invalid data. Missing data is filled in, duplicate data is deduplicated, conflicting data is corrected, and invalid data is removed to form anomaly-handled data.

[0023] The abnormal data is formatted in a unified manner and packaged according to a unified directory structure to obtain standardized data.

[0024] Optionally, obtaining the continuous behavioral representation sequence specifically includes:

[0025] Read standardized data, classify the standardized data at the task level according to the start point, execution process and execution end point of a single robot task, and group the standardized data belonging to the same robot task execution process into the same task unit to obtain task unit data;

[0026] Perform task phase anchoring processing on the data of each task unit, read the task trigger signal, control state change and task log in the data of each task unit, determine the task phase boundary according to the task start node, task progress node and task end node corresponding to the task trigger signal, correct the task phase boundary according to the state switching sequence corresponding to the control state change, verify the task phase boundary according to the task execution record corresponding to the task log, divide the data of each task unit into at least two task phase units, and obtain the task phase unit data corresponding to each task unit;

[0027] Feature extraction is performed on the phase unit data of each task to obtain multi-source behavioral feature data corresponding to each phase unit of the task.

[0028] Perform time-series mapping on the multi-source behavioral feature data corresponding to each task phase unit to obtain the time-series mapping feature data corresponding to each task phase unit;

[0029] The temporal mapping feature data are connected according to the task phase order, and the temporal mapping feature data within the same task unit are continuously combined according to the order of the task phases to form a continuous behavioral representation sequence covering the complete execution process of a single robot task.

[0030] Optionally, determining the semantic condition switching boundary specifically includes:

[0031] The continuous behavior representation sequence is sequentially truncated according to the preset window length and preset sliding step size. A subsequence is constructed for each segment of continuously arranged behavior representation data in the continuous behavior representation sequence, forming a set of subsequences corresponding to the continuous behavior representation sequence.

[0032] The FLUSS algorithm is used to perform nearest neighbor search processing in the FLUSS semantic condition segmentation on the subsequence set. For each subsequence, the remaining subsequences in the subsequence set are selected as comparison objects. The degree of difference between the current subsequence and each comparison object subsequence is calculated one by one. The comparison object subsequence with the smallest degree of difference and satisfying the preset interval condition is determined as the nearest neighbor subsequence of the current subsequence, forming a set of nearest neighbor relationships that correspond one-to-one with each subsequence.

[0033] The arc connection construction process in FLUSS semantic segmentation is performed on the nearest neighbor relationship set. The starting position of each subsequence and the starting position of the corresponding nearest neighbor subsequence are used as the two ends of the connection. Arc connection relationships are established at the temporal positions of the continuous behavior representation sequence. The number of arc connections passing through each temporal position in the continuous behavior representation sequence is counted to form the arc coverage result corresponding to the continuous behavior representation sequence.

[0034] The arc coverage results are processed by FLUSS semantic working condition segmentation. According to the total temporal length of the continuous behavior representation sequence and the relative position of each temporal position in the continuous behavior representation sequence, an ideal arc coverage baseline corresponding to each temporal position is constructed. The actual arc coverage of each temporal position is normalized with the corresponding ideal arc coverage baseline to form an arc correction curve corresponding to the continuous behavior representation sequence.

[0035] The local valley detection process in FLUSS semantic working condition segmentation is performed on the arc correction curve. The correction value of the arc correction curve at each time position is read one by one in the time sequence. The correction value of the current time position is compared with the correction value of the previous time position, the correction value of the next time position, and the preset valley judgment threshold. The time positions that meet the local valley conditions are selected as candidate boundary positions.

[0036] The candidate boundary positions are confirmed by performing boundary confirmation processing. Candidate boundary positions that do not meet the minimum boundary interval condition are eliminated according to their order in the continuous behavior representation sequence. Candidate boundary positions that meet the boundary interval condition are retained as semantic condition switching boundaries, forming a set of semantic condition switching boundaries corresponding to the continuous behavior representation sequence.

[0037] Optionally, obtaining the key action range specifically includes:

[0038] Arrange the semantic working condition switching boundaries according to their temporal position in the continuous behavior representation sequence, and extend each semantic working condition switching boundary forward and backward by a preset range to form a boundary association window that corresponds one-to-one with each semantic working condition switching boundary.

[0039] The relationship between the boundary association windows corresponding to the adjacent semantic working condition switching boundary is determined. The end position of the previous boundary association window and the start position of the next boundary association window are read to determine the separation relationship, connection relationship and overlap relationship between adjacent boundary association windows, and form the boundary association window relationship result.

[0040] For adjacent boundary associated windows that satisfy separation, connection, and overlap relationships, interval determination, interval connection, and interval merging are performed respectively to form a set of key action intervals.

[0041] Optionally, obtaining the pre-labeling results specifically includes:

[0042] The key action intervals are arranged according to their temporal position in the continuous behavior representation sequence, and data segments are extracted from the continuous behavior representation sequence according to the start and end positions of each key action interval to form a set of key action interval data segments.

[0043] Keyframe extraction processing is performed on each data segment in the key action interval data segment set. The grayscale difference value between two adjacent frames is calculated according to the time order of the video frame sequence. Video frames with grayscale difference values ​​greater than the preset keyframe threshold are selected as keyframes to obtain the keyframe set.

[0044] Perform key node segment generation processing on the key action interval data segments corresponding to the key frame set to form a key node segment set;

[0045] Automatic pre-labeling is performed on the key node segment set. Based on the temporal position relationship of the key node segment in the key action interval, the corresponding behavioral feature data is read, and the action category is predicted on the behavioral feature data to obtain the action label result corresponding to each key node segment.

[0046] The key node segment set is associated with the corresponding action label results to form a pre-labeled result containing the location of key action intervals, the location of key node segments, and action label information.

[0047] Optionally, the output of the robot corpus data specifically includes:

[0048] The pre-annotation results are manually reviewed and corrected according to the correspondence between key action intervals, key frames, key node segments and action labels. The action labels are then corrected to obtain the annotation data.

[0049] The labeled data is imported into the behavior model for training. Training samples are constructed according to the correspondence between key action intervals, key node segments and action labels in the labeled data. The behavior model is then trained and its parameters are tuned to obtain the trained behavior model and the model judgment results.

[0050] Extract interval labeling feature information from key action intervals that have been manually verified and confirmed, and then combine them to construct a working condition fingerprint;

[0051] The collected results are matched with the working condition fingerprint, and the collected results are judged in combination with the behavior model judgment results. Supplementary collection judgment is performed on the collected results that do not meet the working condition fingerprint matching conditions, and key review judgment is performed on the collected results that meet the working condition fingerprint matching conditions and require manual confirmation, thus forming supplementary collection results and review results.

[0052] The trained behavior model is connected to a real robot for testing. The test results of the real robot are compared with the verification results. When the test results meet the output conditions, the robot's corpus data is output.

[0053] The beneficial effects of this invention are:

[0054] This invention collects, standardizes, and maps multi-source behavior temporal data from robot corpora, unifying data from different interfaces, sources, and types into a continuous behavior representation sequence. This improves the relevance and processability of robot task process data and reduces the problems of data dispersion, inconsistent structure, and inconsistent temporal sequence that exist when multi-source heterogeneous data is directly used for annotation and training.

[0055] This invention performs FLUSS semantic condition segmentation processing on continuous behavior representation sequences to determine semantic condition switching boundaries and further generate key action intervals. This enables the continuous behavior process of the robot to be divided into intervals according to the behavior change rules, thereby improving the accuracy of key action recognition and the pertinence of key interval extraction.

[0056] This invention performs interval-based annotation processing within key action intervals to complete keyframe extraction, key node segment generation, and automatic pre-annotation. This transforms the annotation process from traditional large-scale manual processing to targeted annotation processing focused on key action intervals, thereby improving the automation level and efficiency of robot corpus acquisition and annotation.

[0057] This invention improves the integrity, consistency, and training usability of robot corpus data by manually reviewing and correcting the pre-annotation results, and by combining behavioral model training, working condition fingerprint construction, and real robot testing. Attached Figure Description

[0058] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0059] Figure 1 This is a flowchart of the data acquisition and annotation method for AI-based data acquisition robots proposed in this invention;

[0060] Figure 2 This is a schematic diagram illustrating the semantic condition segmentation and key action interval generation of the AI-based data acquisition robot corpus acquisition and annotation method proposed in this invention. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0062] refer to Figures 1-2 The data collection and annotation method for AI-based data acquisition robots includes the following steps:

[0063] Collect raw data from the robot's speech corpus to obtain the raw dataset;

[0064] Perform data standardization on the original dataset to obtain standardized data;

[0065] Perform multi-source behavioral temporal mapping processing on standardized data, establish task units and task phase units, and perform feature extraction and temporal mapping to obtain a continuous behavioral representation sequence;

[0066] FLUSS semantic condition segmentation is performed on the continuous behavior representation sequence. Arc connection relationship is constructed based on the nearest neighbor relationship of the subsequence of the continuous behavior representation sequence. The corresponding arc correction curve is calculated, and the semantic condition switching boundary is determined based on the local trough position of the arc correction curve.

[0067] The preset range is extended forward and backward for each semantic working condition switching boundary to form a boundary association window, and key action ranges are generated based on the boundary association window relationship;

[0068] Based on the key action intervals, interval labeling processing is performed. Keyframes are extracted and key node segments are generated within the data segments corresponding to each key action interval. Automatic pre-labeling is then performed to obtain the pre-labeling results.

[0069] The pre-labeled results are manually reviewed and corrected to obtain labeled data. At the same time, the behavior model is trained, the working condition fingerprint is constructed, and the collected results are supplemented and key reviews are performed. The trained behavior model is connected to a real robot for testing, and the robot's corpus data is output.

[0070] In this embodiment, obtaining the original dataset specifically includes:

[0071] By connecting to robot manufacturer platform interfaces, general component underlying call interfaces, and self-developed sensor interfaces, the robot calls the data acquisition channels of different cameras on the robot body to collect video data, status field data, behavior control data, and behavior data during the execution of a single task, forming initial data collection.

[0072] The initial collected data is processed for source and time correspondence, and the initial collected data is mapped to the collection interface and collection channel that generated the data. The data is also arranged in order according to the collection time to form associated collected data.

[0073] Read the task start and end information corresponding to the associated collected data, and perform task-level aggregation of the associated collected data according to the execution process of a single robot task. Combine the data within the same task start and end range into a data set of the same task to obtain the original dataset.

[0074] In this embodiment, obtaining standardized data specifically includes:

[0075] Read the original dataset and classify and organize it according to the data source and task correspondence to form data to be standardized;

[0076] To standardize data, field uniform processing is performed, converting corresponding fields from different data sources into fields with uniform names, uniform order, and uniform types, thus forming field-standardized data.

[0077] The time-standardized data of the fields is uniformly processed by aligning the time according to the collection time of data from various sources within the same task to form time-standardized data.

[0078] Anomaly handling is performed on time-standardized data to identify missing, duplicate, conflicting, and invalid data. Missing data is filled in, duplicate data is deduplicated, conflicting data is corrected, and invalid data is removed to form anomaly-handled data.

[0079] The abnormal data is formatted in a unified manner and packaged according to a unified directory structure to obtain standardized data.

[0080] In this embodiment, obtaining the continuous behavioral representation sequence specifically includes:

[0081] Read standardized data, classify the standardized data at the task level according to the start point, execution process and execution end point of a single robot task, and group the standardized data belonging to the same robot task execution process into the same task unit to obtain task unit data;

[0082] Perform task phase anchoring processing on the data of each task unit, read the task trigger signal, control state change and task log in the data of each task unit, determine the task phase boundary according to the task start node, task progress node and task end node corresponding to the task trigger signal, correct the task phase boundary according to the state switching sequence corresponding to the control state change, verify the task phase boundary according to the task execution record corresponding to the task log, divide the data of each task unit into at least two task phase units, and obtain the task phase unit data corresponding to each task unit;

[0083] Feature extraction is performed on the data of each task phase unit. The feature extraction operation includes extracting video features, state field features, behavior control features and sensor features in each task phase unit. The changes in image content, motion and inter-frame changes in the video data are mapped to video features. The changes in state values ​​and state switching relationships in the state field data are mapped to state field features. The changes in control commands and execution order in the behavior control data are mapped to behavior control features. The changes in sensor output and sensor response in the sensor data are mapped to sensor features, thus obtaining multi-source behavior feature data corresponding to each task phase unit.

[0084] The multi-source behavioral feature data corresponding to each task phase unit is time-series mapped. The time-series mapping includes aligning and associating video features, state field features, behavior control features and sensor features according to the acquisition order and task execution order of various feature data in each task phase unit. Features located in the same task phase unit and in the same time period are mapped to the same time position, and features located in adjacent task phase units and in a sequential relationship are mapped to consecutive time positions, thus obtaining the time-series mapped feature data corresponding to each task phase unit.

[0085] The temporal mapping feature data are connected according to the task phase order, and the temporal mapping feature data within the same task unit are continuously combined according to the order of the task phases to form a continuous behavioral representation sequence covering the complete execution process of a single robot task.

[0086] This invention constructs a continuous behavioral representation sequence covering the complete execution process of a single robot task by performing task-level classification, task phase anchoring, multi-source feature extraction, and temporal mapping on standardized data. This improves the temporal correlation of multi-source heterogeneous data, the completeness of task process expression, and the ability to organize behavioral features, providing a stable data foundation for semantic condition segmentation, key action interval generation, and automatic pre-labeling.

[0087] In this embodiment, the determination of the semantic condition switching boundary specifically includes:

[0088] The continuous behavior representation sequence is sequentially extracted according to the preset window length and preset sliding step size. A subsequence is constructed for each segment of continuously arranged behavior representation data in the continuous behavior representation sequence, forming a set of subsequences corresponding to the continuous behavior representation sequence. Each subsequence in the set of subsequences maintains the original time sequence.

[0089] The FLUSS algorithm is used to perform nearest neighbor search processing in the FLUSS semantic condition segmentation on the subsequence set. For each subsequence, the remaining subsequences in the subsequence set are selected as comparison objects. The degree of difference between the current subsequence and each comparison object subsequence is calculated one by one. The comparison object subsequence with the smallest degree of difference and satisfying the preset interval condition is determined as the nearest neighbor subsequence of the current subsequence, forming a set of nearest neighbor relationships that correspond one-to-one with each subsequence.

[0090] The arc connection construction process in FLUSS semantic segmentation is performed on the nearest neighbor relationship set. The starting position of each subsequence and the starting position of the corresponding nearest neighbor subsequence are used as the two ends of the connection. Arc connection relationships are established at the temporal positions of the continuous behavior representation sequence. The number of arc connections passing through each temporal position in the continuous behavior representation sequence is counted to form the arc coverage result corresponding to the continuous behavior representation sequence.

[0091] The arc coverage results are processed by FLUSS semantic working condition segmentation. According to the total temporal length of the continuous behavior representation sequence and the relative position of each temporal position in the continuous behavior representation sequence, an ideal arc coverage baseline corresponding to each temporal position is constructed. The actual arc coverage of each temporal position is normalized with the corresponding ideal arc coverage baseline to form an arc correction curve corresponding to the continuous behavior representation sequence.

[0092] The local valley detection process in FLUSS semantic working condition segmentation is performed on the arc correction curve. The correction value of the arc correction curve at each time position is read one by one in the time sequence. The correction value of the current time position is compared with the correction value of the previous time position, the correction value of the next time position, and the preset valley judgment threshold. The time positions that meet the local valley conditions are selected as candidate boundary positions.

[0093] The candidate boundary positions are confirmed by performing boundary confirmation processing. Candidate boundary positions that do not meet the minimum boundary interval condition are eliminated according to their order in the continuous behavior representation sequence. Candidate boundary positions that meet the boundary interval condition are retained as semantic condition switching boundaries, forming a set of semantic condition switching boundaries corresponding to the continuous behavior representation sequence.

[0094] This invention employs the FLUSS algorithm to perform semantic condition segmentation on continuous behavior representation sequences, and combines nearest neighbor search, arc connection construction, arc correction, and local valley detection to determine semantic condition switching boundaries. This improves the accuracy of continuous behavior process segmentation, the ability to identify key state transitions, and the stability of boundary localization, providing a reliable foundation for key action interval generation, automatic pre-labeling, and subsequent model training.

[0095] In this embodiment, obtaining the key action range specifically includes:

[0096] Arrange the semantic working condition switching boundaries according to their temporal position in the continuous behavior representation sequence, and extend each semantic working condition switching boundary forward and backward by a preset range to form a boundary association window that corresponds one-to-one with each semantic working condition switching boundary.

[0097] The relationship between the boundary association windows corresponding to the adjacent semantic working condition switching boundary is determined. The end position of the previous boundary association window and the start position of the next boundary association window are read to determine the separation relationship, connection relationship and overlap relationship between adjacent boundary association windows, and form the boundary association window relationship result.

[0098] For adjacent boundary associated windows that satisfy separation, connection, and overlap relationships, interval determination, interval connection, and interval merging are performed respectively to form a set of key action intervals.

[0099] In this embodiment, obtaining the pre-annotation results specifically includes:

[0100] The key action intervals are arranged according to their temporal position in the continuous behavior representation sequence, and data segments are extracted from the continuous behavior representation sequence according to the start and end positions of each key action interval to form a set of key action interval data segments.

[0101] Keyframe extraction processing is performed on each data segment in the key action interval data segment set. The grayscale difference value between two adjacent frames is calculated according to the time order of the video frame sequence. The grayscale difference value is obtained by accumulating the grayscale difference between the current frame image and the previous frame image at the pixel position pixel by pixel. Video frames with grayscale difference values ​​greater than the preset keyframe threshold are selected as keyframes according to the preset keyframe threshold, and a keyframe set is obtained.

[0102] The key node segment generation process is performed on the key action interval data segments corresponding to the key frame set. The key node segment generation process includes grouping the key frames according to the temporal position relationship of the key frames in the key action interval, determining the continuous video frame sequence corresponding to adjacent key frames as key node segments, and recording the start frame position and end frame position of each key node segment to form a key node segment set.

[0103] Automatic pre-labeling is performed on the key node segment set. Based on the temporal position relationship of the key node segment in the key action interval, the corresponding behavioral feature data is read, and the action category is predicted on the behavioral feature data to obtain the action label result corresponding to each key node segment.

[0104] The key node segment set is associated with the corresponding action label results to form a pre-labeled result containing the location of key action intervals, the location of key node segments, and action label information.

[0105] This invention transforms high-value action segments in continuous behavioral representation sequences into structured pre-annotated results by extracting keyframes, generating key node segments, and automatically pre-annotating around key action intervals. This improves the automation level of robot corpus annotation, the accuracy of key action localization, and the completeness of key node extraction, reduces the burden of manual segment-by-segment screening, and improves the efficiency and consistency of manual review, behavioral model training, and corpus construction.

[0106] In this embodiment, the output of the robot corpus data specifically includes:

[0107] The pre-annotation results are manually reviewed and corrected according to the correspondence between key action intervals, key frames, key node segments and action labels. The action labels are then corrected to obtain the annotation data.

[0108] The labeled data is imported into the behavior model for training. Training samples are constructed according to the correspondence between key action intervals, key node segments and action labels in the labeled data. The behavior model is trained and its parameters are tuned to obtain the trained behavior model and model judgment results. The behavior model is trained with labeled data and is used to identify, judge, verify and output training results of the behavior in the robot task execution process.

[0109] Extract interval labeling feature information from the key action intervals that have been manually verified and confirmed, and combine them to construct the working condition fingerprint. The interval labeling feature information includes interval boundary information, key frame position information, key node segment position information, and corresponding action label information.

[0110] The collected results are matched with the working condition fingerprint, and the collected results are judged in combination with the behavior model judgment results. Supplementary collection judgment is performed on the collected results that do not meet the working condition fingerprint matching conditions, and key review judgment is performed on the collected results that meet the working condition fingerprint matching conditions and require manual confirmation, thus forming supplementary collection results and review results.

[0111] The trained behavior model is connected to a real robot for testing. The test results of the real robot are compared with the verification results. When the test results meet the output conditions, the robot corpus data is output. The robot corpus data is a collection of data formed by collecting, cleaning, encapsulating, labeling, training, verifying and exporting data during one or more robot task executions. It includes video resource data, field data, behavior datasets, application data, labeled data, training data and verification test data.

[0112] This invention integrates manual review of pre-annotated results, behavior model training, operational fingerprint construction, collection result judgment, and real robot testing into a unified output link, realizing closed-loop processing of robot corpus data from collection, annotation to training, verification, and export. This can improve the accuracy, consistency, and traceability of robot corpus data, enhance the behavior model's ability to recognize key actions and its adaptability to practical applications, and improve the reliability and engineering usability of the corpus output results.

[0113] Example 1: To verify the feasibility of the invention in practice, it was applied to a scenario of data collection and annotation for a tangible transport robot in a mixed task. In this scenario, the robot needs to continuously complete actions such as target recognition, path approach, posture adjustment, contact grasping, transport and holding, obstacle avoidance correction, target placement, and return to standby. The task process is continuous, the action switching is frequent, and video data, status field data, behavior control data, and behavior data are generated simultaneously during the same task execution.

[0114] The robot is equipped with multiple body cameras and simultaneously connects to the robot manufacturer's platform interface, the underlying calling interface of general components, and the interface of self-developed sensors, forming a raw data acquisition link covering vision, status, control, and sensor feedback. The acquired raw data is first aggregated at the task level, and then enters the data standardization processing flow. Data from different sources is processed to unify fields, align time, handle anomalies, and unify formats, forming standardized data that can be uniformly called. The standardized data continues to enter the multi-source behavior temporal mapping process, extracting video features, status field features, behavior control features, and sensor features based on task units and task phase units, and mapping them into a continuous behavior representation sequence according to the task execution order. Subsequently, the continuous behavior representation sequence is processed by FLUSS semantic condition segmentation to determine the semantic condition switching boundary, and then key action intervals are generated based on the boundary association window. Within the key action interval, key frame extraction, key node segment generation, and automatic pre-annotation are completed. Finally, the manually reviewed annotated data is imported into the behavior model for training, and the trained behavior model is connected to a real robot for testing, thereby outputting robot corpus data.

[0115] In this implementation, raw data acquisition no longer employs a single video recording method, but rather collects data throughout the entire execution process of a single robot task. The robot manufacturer's platform interface outputs the robot's task status, action execution status, and operation logs; the underlying interface of common components outputs control feedback from the execution parts; the self-developed sensor interface outputs external sensor data; and multiple cameras on the robot simultaneously output multi-view video streams. The system binds the above data according to a unified acquisition session, establishing a task identifier at the start of the task and closing the current acquisition session at the end of the task, thereby merging multi-source data from a single complete task into a single raw dataset. After the raw dataset is formed, the system performs data standardization processing, converting data from different sources, with different fields, and different time granularities into a unified field structure and a unified time order, and encapsulating it into standardized data. For video data, the system retains the original frame sequence and time stamps; for status field data and behavior control data, the system retains their state change relationships and control order; for sensor data, the system retains their response changes and sampling order relationships, thus ensuring that subsequent time-series mapping can be associated with a unified task timeline.

[0116] In the multi-source behavior temporal mapping stage, the system first divides the standardized data into task units according to a single robot task. Then, based on the task trigger signal, control state changes, and task logs, the task units are anchored to task phases, refining a single task into multiple task phase units. Subsequently, the system extracts video features, state field features, behavior control features, and sensor features from each task phase unit, and performs temporal mapping according to the order of acquisition and task execution. This ensures that multi-source features within the same task phase and belonging to the same time period are mapped to the same temporal position, and multi-source features within adjacent task phases and in a sequential relationship are mapped to consecutive temporal positions, ultimately forming a continuous behavior representation sequence covering the entire process of a single task. In this way, robot behavior data that was originally scattered across different interfaces and media is reconstructed into a unified continuous behavior expression, laying the foundation for subsequent semantic condition segmentation.

[0117] In the semantic condition segmentation stage, the system uses the FLUSS algorithm to process the continuous behavior representation sequence. First, the continuous behavior representation sequence is sequentially truncated according to the preset window length and sliding step size to construct a set of subsequences. Then, for each subsequence, a nearest neighbor search is performed to determine the comparison object subsequence with the smallest difference and that meets the interval condition as the nearest neighbor subsequence. Furthermore, the relationship between the starting position of the subsequence and the starting position of the nearest neighbor subsequence is used to establish an arc connection relationship, and the number of arc coverages at each time position of the continuous behavior representation sequence is counted to form an arc coverage result. After that, the system constructs an ideal arc coverage baseline based on the total time length of the continuous behavior representation sequence and the relative positions of each time position. Then, the actual arc coverage result is normalized to form an arc correction curve. By performing local trough detection and boundary confirmation on the arc correction curve, the system can determine the semantic condition switching boundary. The semantic working condition switching boundary is not directly used as the annotation output. Instead, it continues to expand forward and backward within a preset range to form a boundary association window. Based on the separation, connection and overlap relationships between adjacent boundary association windows, key action intervals are generated. In this way, the system transforms the originally continuous and lengthy task behavior process into several key action intervals with clear action meaning and annotation value.

[0118] During the interval labeling stage, the system reads key action intervals and extracts corresponding data segments from continuous behavior representation sequences to form a set of key action interval data segments. For each key action interval data segment, the system performs keyframe extraction processing. Specifically, it calculates the grayscale difference between adjacent frames in the video frame sequence, accumulating the grayscale differences pixel by pixel to obtain the grayscale difference value. Then, it selects video frames with grayscale difference values ​​higher than a preset keyframe threshold as keyframes. After keyframe extraction, the system generates key node segments based on the temporal position relationship of key frames within the key action intervals. It identifies the continuous video frame sequences corresponding to adjacent key frames as key node segments and records the start and end frame positions of each key node segment. Subsequently, the system calls its self-developed model, CoreMark R1, to perform automatic pre-labeling processing on the key node segments. CoreMark R1 learns autonomously from the key action data stream and predicts the action category of the behavioral feature data in the key action intervals, outputting the action label results corresponding to the key node segments. Meanwhile, the system can also complete the boundaries of key node segments by combining the grayscale color difference comparison results, thereby forming a pre-annotation result containing key action interval positions, key node segment positions and action label information. The pre-annotation result is not directly used as the final corpus, but enters the manual review and correction process. Manual annotators confirm and revise the action labels, key node segment boundaries and key action interval positions to obtain annotated data that can be used for training.

[0119] During the training and output phases, the system imports labeled data into the behavior model training process, constructs training samples according to the correspondence between key action intervals, key node segments, and action labels, trains and tunes the behavior model, and obtains the trained behavior model and model judgment results. For key action intervals confirmed by manual review, the system further extracts interval labeling feature information and performs association and combination to construct a working condition fingerprint. Then, the system matches the newly collected results with the working condition fingerprint and judges the collected results in conjunction with the behavior model judgment results. Supplementary collection judgment is performed on collected results that do not meet the working condition fingerprint matching conditions, and key review judgment is performed on collected results requiring manual confirmation. The trained behavior model is connected to a real robot for testing. When the real robot performs similar tasks, the system compares the test results with the manually reviewed results. When the test results meet the output conditions, the robot corpus data is output. The output robot corpus data includes video resource data, field data, behavior datasets, application data, labeled data, training data, and validation test data, forming a deliverable, reusable, and continuously trainable and optimized dataset.

[0120] To quantitatively illustrate the effectiveness of this invention in this scenario, the traditional manual segment-by-segment screening scheme was used as a control scheme, and the scheme of this invention was used as the implementation scheme. The two schemes were compared under the same task scale, the same robot platform, and the same sensor configuration. In the traditional manual segment-by-segment screening scheme, the original video and status data were collected and saved separately. A human then manually reviewed the long-term video segment by segment, manually determined keyframes and key action boundaries, and then manually compiled action labels. In the scheme of this invention, the original data underwent standardization, multi-source behavior temporal mapping, FLUSS semantic condition segmentation, key action interval generation, and automatic pre-annotation, followed by manual review and correction. The comparison results are shown in Table 1.

[0121] Table 1. Performance Comparison of Different Robot Corpus Construction Schemes

[0122] Evaluation indicators Traditional manual screening methods Invention Solution Original task sample size 1200 segments 1200 segments Total duration of original video 186 hours 186 hours Data standardization processing time 21.8 hours 9.3 hours Time consumption of continuous behavior segmentation 19.1 hours 4.8 hours Total time for a complete round of annotation 97.2 hours 30.4 hours Key action range hit rate 70.8% 91.9% Action category recognition accuracy 82.8% 91.6% Final deliverable corpus proportion 65.3% 91.1%

[0123] Table 1 shows that, under the condition that the number of original task samples and the total duration of the original video remain the same, the proposed solution is significantly lower than the traditional manual screening solution in terms of data standardization processing, continuous behavior segmentation, and the total time for a complete round of annotation. Specifically, the time for data standardization processing is reduced from 21.8 hours to 9.3 hours, the time for continuous behavior segmentation is reduced from 19.1 hours to 4.8 hours, and the total time for a complete round of annotation is reduced from 97.2 hours to 30.4 hours, indicating that the proposed solution can significantly improve the efficiency of robot corpus construction. At the same time, the key action interval hit rate of the proposed solution is increased from 70.8% to 91.9%, the action category recognition accuracy is increased from 82.8% to 91.6%, and the final deliverable corpus ratio is increased from 65.3% to 91.1%, indicating that the proposed solution can not only improve the accuracy of key action recognition and behavior annotation, but also significantly improve the training usability and final delivery quality of robot corpus data.

[0124] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for data acquisition and annotation of language corpora based on artificial intelligence, characterized in that, Includes the following steps: Collect raw data from the robot's speech corpus to obtain the raw dataset; Perform data standardization on the original dataset to obtain standardized data; Perform multi-source behavioral temporal mapping processing on standardized data, establish task units and task phase units, and perform feature extraction and temporal mapping to obtain a continuous behavioral representation sequence; FLUSS semantic condition segmentation is performed on the continuous behavior representation sequence. Arc connection relationship is constructed based on the nearest neighbor relationship of the subsequence of the continuous behavior representation sequence. The corresponding arc correction curve is calculated, and the semantic condition switching boundary is determined based on the local trough position of the arc correction curve. The preset range is extended forward and backward for each semantic working condition switching boundary to form a boundary association window, and key action ranges are generated based on the boundary association window relationship; Based on the key action intervals, interval labeling processing is performed. Keyframes are extracted and key node segments are generated within the data segments corresponding to each key action interval. Automatic pre-labeling is then performed to obtain the pre-labeling results. The pre-labeled results are manually reviewed and corrected to obtain labeled data. At the same time, the behavior model is trained, the working condition fingerprint is constructed, and the collected results are supplemented and key reviews are performed. The trained behavior model is connected to a real robot for testing, and the robot's corpus data is output.

2. The method for collecting and annotating corpus data for an AI-based data acquisition robot according to claim 1, characterized in that, The acquisition of the original dataset specifically includes: By connecting to robot manufacturer platform interfaces, general component underlying call interfaces, and self-developed sensor interfaces, the robot calls the data acquisition channels of different cameras on the robot body to collect video data, status field data, behavior control data, and behavior data during the execution of a single task, forming initial data collection. The initial collected data is processed for source and time correspondence, and the initial collected data is mapped to the collection interface and collection channel that generated the data. The data is also arranged in order according to the collection time to form associated collected data. Read the task start and end information corresponding to the associated collected data, and perform task-level aggregation of the associated collected data according to the execution process of a single robot task. Combine the data within the same task start and end range into a data set of the same task to obtain the original dataset.

3. The method for collecting and annotating corpus data for an AI-based data acquisition robot according to claim 1, characterized in that, The standardization data is obtained specifically in the following ways: Read the original dataset and classify and organize it according to the data source and task correspondence to form data to be standardized; To standardize data, field uniform processing is performed, converting corresponding fields from different data sources into fields with uniform names, uniform order, and uniform types, thus forming field-standardized data. The time-standardized data of the fields is uniformly processed by aligning the time according to the collection time of data from various sources within the same task to form time-standardized data. Anomaly handling is performed on time-standardized data to identify missing, duplicate, conflicting, and invalid data. Missing data is filled in, duplicate data is deduplicated, conflicting data is corrected, and invalid data is removed to form anomaly-handled data. The abnormal data is formatted in a unified manner and packaged according to a unified directory structure to obtain standardized data.

4. The method for collecting and annotating corpus data of an AI-based data acquisition robot according to claim 1, characterized in that, The specific steps to obtain the continuous behavioral representation sequence include: Read standardized data, classify the standardized data at the task level according to the start point, execution process and execution end point of a single robot task, and group the standardized data belonging to the same robot task execution process into the same task unit to obtain task unit data; Perform task phase anchoring processing on the data of each task unit, read the task trigger signal, control state change and task log in the data of each task unit, determine the task phase boundary according to the task start node, task progress node and task end node corresponding to the task trigger signal, correct the task phase boundary according to the state switching sequence corresponding to the control state change, verify the task phase boundary according to the task execution record corresponding to the task log, divide the data of each task unit into at least two task phase units, and obtain the task phase unit data corresponding to each task unit; Feature extraction is performed on the phase unit data of each task to obtain multi-source behavioral feature data corresponding to each phase unit of the task. Perform time-series mapping on the multi-source behavioral feature data corresponding to each task phase unit to obtain the time-series mapping feature data corresponding to each task phase unit; The temporal mapping feature data are connected according to the task phase order, and the temporal mapping feature data within the same task unit are continuously combined according to the order of the task phases to form a continuous behavioral representation sequence covering the complete execution process of a single robot task.

5. The method for collecting and annotating corpus data for an AI-based data acquisition robot according to claim 1, characterized in that, The determination of the semantic condition switching boundary specifically includes: The continuous behavior representation sequence is sequentially truncated according to the preset window length and preset sliding step size. A subsequence is constructed for each segment of continuously arranged behavior representation data in the continuous behavior representation sequence, forming a set of subsequences corresponding to the continuous behavior representation sequence. The FLUSS algorithm is used to perform nearest neighbor search processing in the FLUSS semantic condition segmentation on the subsequence set. For each subsequence, the remaining subsequences in the subsequence set are selected as comparison objects. The degree of difference between the current subsequence and each comparison object subsequence is calculated one by one. The comparison object subsequence with the smallest degree of difference and satisfying the preset interval condition is determined as the nearest neighbor subsequence of the current subsequence, forming a set of nearest neighbor relationships that correspond one-to-one with each subsequence. The arc connection construction process in FLUSS semantic segmentation is performed on the nearest neighbor relationship set. The starting position of each subsequence and the starting position of the corresponding nearest neighbor subsequence are used as the two ends of the connection. Arc connection relationships are established at the temporal positions of the continuous behavior representation sequence. The number of arc connections passing through each temporal position in the continuous behavior representation sequence is counted to form the arc coverage result corresponding to the continuous behavior representation sequence. The arc coverage results are processed by FLUSS semantic working condition segmentation. According to the total temporal length of the continuous behavior representation sequence and the relative position of each temporal position in the continuous behavior representation sequence, an ideal arc coverage baseline corresponding to each temporal position is constructed. The actual arc coverage of each temporal position is normalized with the corresponding ideal arc coverage baseline to form an arc correction curve corresponding to the continuous behavior representation sequence. The local valley detection process in FLUSS semantic working condition segmentation is performed on the arc correction curve. The correction value of the arc correction curve at each time position is read one by one in the time sequence. The correction value of the current time position is compared with the correction value of the previous time position, the correction value of the next time position, and the preset valley judgment threshold. The time positions that meet the local valley conditions are selected as candidate boundary positions. The candidate boundary positions are confirmed by performing boundary confirmation processing. Candidate boundary positions that do not meet the minimum boundary interval condition are eliminated according to their order in the continuous behavior representation sequence. Candidate boundary positions that meet the boundary interval condition are retained as semantic condition switching boundaries, forming a set of semantic condition switching boundaries corresponding to the continuous behavior representation sequence.

6. The method for collecting and annotating corpus data for an AI-based data acquisition robot according to claim 1, characterized in that, The key action range is obtained specifically by: Arrange the semantic working condition switching boundaries according to their temporal position in the continuous behavior representation sequence, and extend each semantic working condition switching boundary forward and backward by a preset range to form a boundary association window that corresponds one-to-one with each semantic working condition switching boundary. The relationship between the boundary association windows corresponding to the adjacent semantic working condition switching boundary is determined. The end position of the previous boundary association window and the start position of the next boundary association window are read to determine the separation relationship, connection relationship and overlap relationship between adjacent boundary association windows, and form the boundary association window relationship result. For adjacent boundary associated windows that satisfy separation, connection, and overlap relationships, interval determination, interval connection, and interval merging are performed respectively to form a set of key action intervals.

7. The method for collecting and annotating corpus data for an AI-based data acquisition robot according to claim 1, characterized in that, The pre-annotation results are obtained specifically in the following ways: The key action intervals are arranged according to their temporal position in the continuous behavior representation sequence, and data segments are extracted from the continuous behavior representation sequence according to the start and end positions of each key action interval to form a set of key action interval data segments. Keyframe extraction processing is performed on each data segment in the key action interval data segment set. The grayscale difference value between two adjacent frames is calculated according to the time order of the video frame sequence. Video frames with grayscale difference values ​​greater than the preset keyframe threshold are selected as keyframes to obtain the keyframe set. Perform key node segment generation processing on the key action interval data segments corresponding to the key frame set to form a key node segment set; Automatic pre-labeling is performed on the set of key node segments. Based on the temporal position relationship of the key node segments in the key action interval, the corresponding behavioral feature data is read, and the action category is predicted on the behavioral feature data to obtain the action label result corresponding to each key node segment. The key node segment set is associated with the corresponding action label results to form a pre-labeled result containing the location of key action intervals, the location of key node segments, and action label information.

8. The method for collecting and annotating corpus data of an AI-based data acquisition robot according to claim 1, characterized in that, The output of the robot corpus data specifically includes: The pre-annotation results are manually reviewed and corrected according to the correspondence between key action intervals, key frames, key node segments and action labels. The action labels are then corrected to obtain the annotation data. The labeled data is imported into the behavior model for training. Training samples are constructed according to the correspondence between key action intervals, key node segments and action labels in the labeled data. The behavior model is then trained and its parameters are tuned to obtain the trained behavior model and the model judgment results. Extract interval labeling feature information from key action intervals that have been manually verified and confirmed, and then combine them to construct a working condition fingerprint; The collected results are matched with the working condition fingerprint, and the collected results are judged in combination with the behavior model judgment results. Supplementary collection judgment is performed on the collected results that do not meet the working condition fingerprint matching conditions, and key review judgment is performed on the collected results that meet the working condition fingerprint matching conditions and require manual confirmation, thus forming supplementary collection results and review results. The trained behavior model is connected to a real robot for testing. The test results of the real robot are compared with the verification results. When the test results meet the output conditions, the robot's corpus data is output.

Images