A multi-modal data labeling method and system

Abstract

The application provides a multi-modal data labeling method and system, which is suitable for the technical field of data processing. The method comprises the following steps: obtaining to-be-labeled region coordinate information according to to-be-labeled multi-modal source data, to-be-labeled sample dimension information, to-be-labeled range information and historical labeling path information; generating current labeling path information according to to-be-labeled label system information, to-be-labeled range information and to-be-labeled region coordinate information; and generating data labeling information according to the current labeling path information, to-be-labeled label system information, to-be-labeled target recognition model and a preset data labeling parameter set. The application effectively improves the positioning accuracy, path rationality and execution consistency of multi-modal data labeling, ensures the accuracy of multi-modal data labeling, and significantly improves the overall efficiency and standardization level of multi-modal data labeling.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to multimodal data annotation methods and systems. Background Technology

[0002] With the rapid popularization of multimodal large models and cross-modal perception technologies, the application of heterogeneous data such as text, images, audio, video, and 3D point clouds continues to deepen, and multimodal data annotation has become a core supporting link for artificial intelligence model training.

[0003] In existing technologies, the implementation path is mostly based on AI pre-labeling combined with manual correction. This involves generating preliminary labels and selecting regions for multimodal source data through pre-trained models, completing basic labeling according to a fixed labeling system, and relying on manual review and correction of the labeling results to achieve the labeling of multimodal data.

[0004] However, existing technologies suffer from technical problems such as insufficient accuracy in locating labeled areas, weak adaptability to multimodal data, low level of intelligence in the labeling process, and difficulty in coordinating labeling accuracy and efficiency. Summary of the Invention

[0005] In view of this, embodiments of this application provide a multimodal data annotation method and system, which aims to solve the problems existing in the prior art, such as inaccurate annotation area positioning, unreasonable annotation path planning, insufficient multimodal data adaptation capability, low level of intelligence in the annotation process, and inability to balance annotation accuracy and annotation efficiency.

[0006] The first aspect of this application provides a multimodal data annotation method, including:

[0007] Acquire the multimodal source data to be labeled, the dimensional information of the samples to be labeled, the label system information to be labeled, the label range information, the historical labeling path information, and the target recognition model to be labeled;

[0008] Based on the multimodal source data to be labeled, the dimensional information of the sample to be labeled, the range information to be labeled, and the historical labeling path information, the coordinate information of the region to be labeled is obtained;

[0009] Based on the information of the label system to be labeled, the information of the range to be labeled, and the coordinate information of the area to be labeled, the current labeling path information is generated;

[0010] Data annotation information is generated based on the current annotation path information, the label system information to be annotated, the target recognition model to be annotated, and the preset data annotation parameter set.

[0011] A second aspect of this application provides a multimodal data annotation system, comprising:

[0012] The information acquisition module is used to acquire the multimodal source data to be labeled, the dimensional information of the samples to be labeled, the label system information to be labeled, the range information to be labeled, the historical labeling path information, and the target recognition model to be labeled.

[0013] The module for generating coordinate information of the region to be labeled is used to obtain the coordinate information of the region to be labeled based on the multimodal source data to be labeled, the dimensional information of the sample to be labeled, the range information to be labeled, and the historical labeling path information.

[0014] The current annotation path information generation module is used to generate current annotation path information based on the label system information to be annotated, the annotation range information, and the coordinate information of the area to be annotated;

[0015] The data annotation information generation module is used to generate data annotation information based on the current annotation path information, the label system information to be annotated, the target recognition model to be annotated, and the preset data annotation parameter set.

[0016] A third aspect of this application provides a terminal device, the terminal device including a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the computer program to implement the steps of the multimodal data annotation method described in the first aspect above.

[0017] A fourth aspect of this application provides a computer-readable storage medium, comprising: storing a computer program, wherein when executed by a processor, the computer program implements the steps of the multimodal data annotation method described in the first aspect above.

[0018] The beneficial effects of this application embodiment compared with the prior art are: this application accurately adapts to the individual differences of the multimodal source data to be labeled and the dynamic requirements of multimodal labeling, realizes high-precision labeling of multimodal data, and makes the generated data labeling information more in line with the actual labeling requirements, thereby improving the positioning accuracy, labeling path matching degree and labeling information consistency of multimodal data labeling, so as to meet the actual needs of high-precision and large-scale labeling of multimodal data in artificial intelligence model training. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1This is a schematic diagram illustrating the implementation process of the multimodal data annotation method provided in Embodiment 1 of this application;

[0021] Figure 2 This is a schematic diagram illustrating the implementation process of the multimodal data annotation method provided in Embodiment 2 of this application;

[0022] Figure 3 This is a schematic diagram illustrating the implementation process of the multimodal data annotation method provided in Embodiment 3 of this application;

[0023] Figure 4 This is a schematic diagram illustrating the implementation process of the multimodal data annotation method provided in Embodiment 4 of this application;

[0024] Figure 5 This is a schematic diagram illustrating the implementation process of the multimodal data annotation method provided in Embodiment 5 of this application;

[0025] Figure 6 This is a schematic diagram illustrating the implementation process of the multimodal data annotation method provided in Embodiment Six of this application;

[0026] Figure 7 This is a schematic diagram illustrating the implementation process of the multimodal data annotation method provided in Embodiment 7 of this application;

[0027] Figure 8 This is a schematic diagram of the structure of the multimodal data annotation system provided in the embodiments of this application;

[0028] Figure 9 This is a schematic diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0030] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0031] Figure 1 The implementation flowchart of the multimodal data annotation method provided in Embodiment 1 of this application is shown, and is described in detail below:

[0032] Step S101: Obtain the multimodal source data to be labeled, the dimensional information of the sample to be labeled, the label system information to be labeled, the range information to be labeled, the historical labeling path information, and the target recognition model to be labeled.

[0033] In this embodiment, the multimodal source data to be labeled can be a heterogeneous data set such as text, images, audio, and video required for training artificial intelligence models, providing the original data foundation for labeling. The multimodal source data to be labeled can be acquired in batches through data acquisition devices or imported into a preset multimodal dataset through a data management platform. The dimensionality information of the samples to be labeled refers to the sample attribute parameters corresponding to the multimodal source data, serving as the basis for determining the range of the area to be labeled, avoiding deviations in the labeled area due to differences in sample dimensions. The dimensionality information of the samples to be labeled can be obtained after extracting attributes from the multimodal source data using data parsing algorithms. The label system information to be labeled can be a set of labels and classification rules used for labeling multimodal data, representing the core standard for multimodal data labeling. The label system information to be labeled can be obtained by importing a manually preset label system file through a label management terminal. The range information to be labeled refers to the preset specific area parameters in the multimodal source data to be labeled, clarifying the boundaries and range of the labeling, avoiding exceeding the preset area and affecting the accuracy of the labeling. The range information to be labeled can be obtained by timely capturing the coordinate data of the multimodal data labeling area through data positioning software or devices. Historical annotation path information refers to the annotation path data from previous annotations of similar multimodal data. This data is used to analyze path deviation patterns and assist in optimizing the current annotation path. The system continuously records the path data for each annotation, updating the previous path information daily to generate a historical record. The target recognition model can be a deep learning-based multimodal target localization and recognition model. It accurately matches the positional relationship between the multimodal source data to be annotated and the region to be annotated, thereby achieving accurate localization of multimodal data annotations. This target recognition model can be trained on a large dataset of multimodal source data, corresponding labels to be annotated, and annotation paths. It can be implemented using a CNN+Attention hybrid architecture or a multi-Transformer hybrid architecture. Data augmentation algorithms are introduced during training to optimize localization accuracy, ensuring accurate localization even under varying data deviations.

[0034] In this embodiment, the multimodal source data to be labeled may include text data information, image pixel information, audio spectrum information, video frame sequence information, and 3D point cloud data information; the dimensional information of the samples to be labeled may include sample category information, sample resolution information, sample data volume information, and sample heterogeneity information; the label system information to be labeled may include label classification information, label hierarchy information, and label association rule information; the label range information to be labeled may include the starting coordinates of the region to be labeled, the ending coordinates of the region to be labeled, and the area of ​​the region to be labeled; the historical labeling path information may include historical labeling path deviation information, historical labeling repetition accuracy information, and historical labeling alignment correction information.

[0035] Among them, text data information can refer to the text content and semantic information to be labeled, which is an important component of multimodal annotation, such as various documents and dialogue text; image pixel information can refer to the pixel distribution, color parameters, and other data of the image to be labeled, used for the annotation of image-type targets; audio spectrum information can refer to the frequency, amplitude, and other parameters of the audio to be labeled, used for the annotation of audio-type targets; video frame sequence information can refer to the continuous frame image data of the video to be labeled, used for the temporal annotation of video-type targets; and 3D point cloud data information can refer to the point cloud coordinate distribution data of the 3D object to be labeled, used for the spatial mapping of 3D targets. Note: Sample category information refers to the category to which the sample to be labeled belongs, such as people, objects, scenes, etc., used for classification labeling; sample resolution information refers to the resolution parameters of the image or video to be labeled, affecting the detail accuracy of the labeling; sample data volume information refers to the total amount of multimodal source data to be labeled, used for planning the labeling process; sample heterogeneity information refers to the degree of difference between different types of data in the multimodal data to be labeled, used to adapt to different labeling strategies; label classification information refers to the specific classification of the label, such as entity labels, attribute labels, relationship labels, etc.; label hierarchy information refers to the hierarchical structure of the labels, such as... Primary and secondary labels are used to standardize annotation logic; label association rules refer to the relationships between different labels to ensure annotation consistency; the starting coordinates of the region to be annotated refer to the starting coordinates of the region to be annotated, serving as the reference starting point of the annotation path; the ending coordinates of the region to be annotated refer to the ending coordinates of the region to be annotated, clarifying the boundary endpoint of the annotation; the area information of the region to be annotated refers to the specific area parameters of the region to be annotated, used to determine the size of the annotation range; historical annotation path deviation information refers to the deviation data between past annotation paths and the ideal path, used to analyze deviation patterns; historical annotation repetition accuracy information refers to the positional consistency data of multiple annotations of the same multimodal data, reflecting the stability of the annotation; historical annotation alignment correction information refers to the correction parameters used for path deviation in the past, used to assist in current path optimization; model architecture parameter information refers to the network structure parameters of the target recognition model to be annotated, determining the model's localization performance; model training iteration information refers to the number of iterations, loss values, and other data during model training, reflecting the model's training effect; model localization accuracy information refers to the model's localization accuracy for multimodal targets, used to evaluate model performance.Among them, various data information in the multimodal source data to be labeled can be collected collaboratively by data acquisition devices and data parsing modules; the dimensional information of the samples to be labeled can be obtained by extracting attributes from the multimodal source data through data parsing algorithms; the label system information to be labeled can be obtained by importing a preset label system file through the label management terminal, and obtaining label classification, hierarchy and association rule information after rule parsing; the label range information can be obtained by capturing the preset area of ​​multimodal data through data positioning devices, and obtaining the start and end coordinates and area information through coordinate calculation; the historical labeling path information can be obtained by automatically recording the path, deviation and correction data of each labeling by the system, and updating the newly collected path information with the existing historical data every day to generate a continuous historical record.

[0036] Step S102: Based on the multimodal source data to be labeled, the dimension information of the sample to be labeled, the range information to be labeled, and the historical labeling path information, obtain the coordinate information of the region to be labeled.

[0037] In this embodiment, the multimodal source data and sample dimensional information to be labeled are first encoded to generate basic feature vectors, which serve as the core basis for determining the coordinate information of the region to be labeled. Then, combined with the range information to be labeled, a threshold discrimination and data registration algorithm is used to initially determine the approximate coordinate range of the region to be labeled, avoiding positioning errors caused by differences in sample dimensionality and range definition deviations. Then, based on historical labeling path information and combined with the standard coordinate parameters of similar multimodal samples, a dynamic positioning feature space can be constructed using an LSTM+Transformer hybrid temporal architecture. This space includes key features such as historical positioning deviations and path correction parameters. A collaborative filtering algorithm is used to calculate the positioning similarity between the current multimodal sample and historical similar samples. By referring to the positioning patterns of highly similar samples, the current positioning judgment is optimized, and the preliminary positioning coordinate range is corrected, thereby generating accurate coordinate information of the region to be labeled, ensuring that the coordinate information of the region to be labeled highly matches the actual situation of the multimodal source data to be labeled.

[0038] Step S103: Generate the current annotation path information based on the label system information to be annotated, the annotation range information, and the coordinate information of the area to be annotated.

[0039] In this embodiment, it is understood that the information on the label system to be labeled, the information on the range to be labeled, and the coordinate information of the area to be labeled are all closely related to the generation of the current labeling path information. It can be based on manually preset path generation rules, matching the information on the label system to be labeled with the coordinate information of the area to be labeled to clarify the specific labeling position of the label within the area to be labeled, avoiding labeling outside the area to be labeled. For example, if the coordinate information of the area to be labeled displays preset values ​​for the start and end coordinates, and the information on the label system to be labeled is a set of labels of a specific category, then the labeling positions corresponding to each type of label are aligned with the coordinates of the area to be labeled. Then, combined with the information on the range to be labeled, the boundary constraints of each type of label within the area are determined. Furthermore, referring to the optimization parameters in the historical labeling path information, the labeling path is adjusted to avoid path overlap or omission. Then, a path planning algorithm is used to generate a labeling path that fits the label system to be labeled and adapts to the range of the area to be labeled, thereby generating the current labeling path information, ensuring that the current labeling path information can accurately cover the area to be labeled and meet the labeling requirements of the label system to be labeled.

[0040] Step S104: Generate data annotation information based on the current annotation path information, the label system information to be annotated, the target recognition model to be annotated, and the preset data annotation parameter set.

[0041] In this embodiment, the preset data annotation parameter set can be manually preset and includes a dataset containing core annotation parameters such as annotation accuracy, annotation speed, and annotation confidence. This dataset is used to adapt to the annotation requirements of different multimodal data and label systems. The target recognition model can be a deep learning-based precise localization model used to calibrate the matching degree between the current annotation path information and the label system information in real time. The process involves inputting the current annotation path information and the label system information into the target recognition model, encoding the label system information and path information using a BERT pre-trained model, and then extracting a contextual feature vector after processing with a self-attention mechanism. This vector reflects the matching status between the path and the label. Using this feature vector as a control condition, combined with the preset data annotation parameter set, the multimodal data annotation parameters are adaptively adjusted. For example, the annotation accuracy parameter is adjusted based on the sample resolution information in the dimensional information of the sample to be annotated, and the annotation speed and confidence parameters are adjusted based on the label hierarchy information in the label system information. Then, a path optimization algorithm is used to fine-tune the current annotation path information to ensure that the path and label annotation requirements are fully aligned, thereby generating data annotation information containing key parameters such as annotation path, annotation accuracy, and annotation speed.

[0042] The multimodal data annotation method provided in this application accurately adapts to the individual differences of the multimodal source data to be annotated and the dynamic requirements of multimodal annotation, achieving high-precision annotation of multimodal data. This makes the generated data annotation information more in line with actual annotation requirements, thereby improving the positioning accuracy, annotation path matching degree and annotation information consistency of multimodal data annotation, so as to meet the actual needs of high-precision and large-scale annotation of multimodal data in artificial intelligence model training.

[0043] Figure 2 The flowchart illustrating the implementation of the multimodal data annotation method provided in Embodiment 2 of this application is shown. The difference between this method and Embodiment 1 is that step S104 specifically includes:

[0044] Step S201: Based on the preset feature extraction rules for the target to be labeled, generate multiple benchmark labeled target feature information according to the preset data labeling parameter set.

[0045] In this embodiment, the preset feature extraction rules for the target to be labeled can be manually preset. These rules can refer to a standardized process for processing labeled target data through multimodal feature parsing, feature enhancement, and feature filtering, used to accurately extract the core features of the labeled target. Multiple sets of labeling parameters adapted to different multimodal data can be selected from a preset set of data labeling parameters. Then, for each set of parameters corresponding to the standard labeled target, a multimodal feature parsing algorithm is sequentially executed to obtain the target's core features. A feature enhancement algorithm is then used to optimize feature accuracy. Finally, a feature filtering algorithm is used to remove redundant features, generating baseline labeled target feature information corresponding to each set of parameters. This baseline serves as the standard input feature for subsequent training of the target recognition model, ensuring that the training process has a clear reference benchmark.

[0046] Step S202: Perform feature extraction processing on the multimodal source data to be labeled to obtain multiple target feature information to be matched.

[0047] In this embodiment, the multimodal source data to be labeled includes text data information, image pixel information, audio spectrum information, video frame sequence information, and 3D point cloud data information. The feature extraction process is as follows: First, the various types of data in the multimodal source data to be labeled are converted into structured numerical data, such as text semantic vectors, image pixel coordinates, audio frequency parameters, video frame feature values, and 3D point cloud coordinates. Then, these numerical data are input into a large language model to generate corresponding multimodal target feature description data, such as "the text semantics are clear, the image pixel distribution is uniform, the audio spectrum is stable, and it meets the standard specifications for labeled targets". Then, the multimodal target feature description data can be input into the CNN pre-trained model in the target recognition model to be labeled. After processing through the self-attention mechanism, the context vector is extracted as the feature information of the target to be matched. Since the multimodal source data to be labeled may have different formats and different precisions, multiple target feature information adapted to different working conditions will be generated.

[0048] Step S203: Based on the multiple benchmark labeled target feature information and the multiple target feature information to be matched, train the target recognition model to be labeled to obtain the trained target recognition model.

[0049] In this embodiment, the target recognition model can be a CNN+Attention hybrid architecture model based on deep learning. The training process can be as follows: First, the encoder in the VAE pre-trained model maps the generated target feature information of multiple benchmark labeled targets to low-dimensional latent feature vectors. Then, the low-dimensional latent feature vectors and multiple target feature information to be matched are used as input to the target recognition model. During the training process, the matching error between the target feature information to be matched and the benchmark labeled target feature information is calculated using the benchmark as the standard. The model parameters are updated through the backpropagation algorithm. Then, the data augmentation algorithm is introduced to simulate the changes in target features under different data deviations and different environmental conditions to optimize the model's adaptability. The training process is repeated until the error converges to obtain the trained target recognition model, which is used to accurately identify the positional association between the multimodal source data to be labeled and the region to be labeled, thereby improving the labeling and localization accuracy.

[0050] Step S204: Generate data annotation information based on the current annotation path information, the label system information to be annotated, and the trained target recognition model to be annotated.

[0051] In this embodiment, the current annotation path information and the label system information to be annotated can be input together into the trained target recognition model. The trained target recognition model will first perform feature matching on the current annotation path information and the label system information to be annotated, identify the deviations between the path and the label annotation requirements, and then fine-tune the current annotation path information based on the label system information to be annotated to correct path offset, overlap or omission. Then, combined with the preset data annotation parameter set, according to the target feature information to be matched and the dimension information of the sample to be annotated, the annotation accuracy, annotation speed and other parameters are adjusted. Then, the fine-tuned current annotation path information and the adjusted annotation parameters are integrated to generate data annotation information containing accurate annotation path and adaptive parameters, so as to ensure that multimodal data annotation can accurately fit the requirements of the label system to be annotated and improve the accuracy and consistency of annotation.

[0052] The multimodal data annotation method provided in this application enables the target recognition model to accurately learn the correlation between the multimodal source data to be annotated and the data annotation parameters, improves the adaptability of the data annotation information to the actual needs and sample characteristics of the multimodal source data to be annotated, thereby improving the positioning accuracy and annotation effect of multimodal data annotation, and adapting to the multimodal data annotation needs of different working conditions in artificial intelligence model training.

[0053] Figure 3 The flowchart illustrating the implementation of the multimodal data annotation method provided in Embodiment 3 of this application is shown. Its difference from Embodiment 2 described above lies in:

[0054] The multiple benchmark annotation target feature information includes first benchmark annotation target feature information and second benchmark annotation target feature information;

[0055] The plurality of target feature information to be matched includes first target feature information to be matched and second target feature information to be matched;

[0056] Step S203 specifically includes:

[0057] Step S301: Perform fusion processing based on the first benchmark labeled target feature information and the first target feature information to be matched to generate first target feature fusion information.

[0058] In this embodiment, the first benchmark labeled target feature information is the preset standard labeled target core feature, and the first target feature information to be matched is the target feature extracted from the multimodal source data to be labeled and adapted to specific working conditions. Both are core input data for training the target recognition model. The first benchmark labeled target feature information and the first target feature information to be matched can be standardized first to eliminate biases caused by different feature dimensions. Then, a feature concatenation algorithm is used to fuse the two types of feature information, integrating their core feature elements. Next, feature normalization is used to optimize the accuracy of the fused features and remove redundant interference components. Finally, the validity of the fused feature information is verified to ensure that the fused features accurately reflect the core attributes of the labeled target, thereby generating the first target feature fusion information, laying the foundation for subsequent feature association processing.

[0059] Step S302: Perform fusion processing based on the second reference labeled target feature information and the second target feature information to be matched to generate second target feature fusion information.

[0060] In this embodiment, the second benchmark annotation target feature information and the first benchmark annotation target feature information are standard annotation target features corresponding to different parameters, and the second target feature information to be matched and the first target feature information to be matched are target features to be annotated adapted to different working conditions. The second benchmark annotation target feature information and the second target feature information to be matched can be standardized first to unify feature dimensions and numerical ranges. Then, the same feature concatenation algorithm as in step S301 is used for fusion processing to integrate the key information of the two types of features. Next, a feature enhancement algorithm is used to enhance the recognizability of the fused features, highlighting the core features of the annotated target. Finally, the fused feature information is verified to eliminate invalid feature interference, thereby generating second target feature fusion information, which complements the first target feature fusion information and enriches the feature expression of the annotated target.

[0061] Step S303: Perform association processing based on the first target feature fusion information and the second target feature fusion information to generate target feature cross-association information.

[0062] In this embodiment, the first target feature fusion information and the second target feature fusion information correspond to the labeled target features under different annotation parameters and working conditions, respectively. They have a certain intrinsic relationship and can jointly reflect the complete features of the labeled target. Based on a pre-defined feature association rule, which defines the association logic and calculation method of the two types of fused features, the first and second target feature fusion information are cross-compared to identify their common and differing features. Then, the association degree of the two types of features is calculated using an association algorithm to quantify their intrinsic connection. Finally, the common features, differing features, and association degree data are integrated, and the integrated feature information is optimized to remove duplicate feature components, thereby generating target feature cross-association information. This comprehensively and accurately reflects the feature attributes of the labeled target, providing a more comprehensive feature input for training the target recognition model.

[0063] Step S304: Based on the target feature cross-correlation information, train the target recognition model to be labeled to obtain the trained target recognition model.

[0064] In this embodiment, the target feature cross-correlation information integrates the core features and relationships of the labeled targets under different parameters and working conditions, and serves as the core input for training the target recognition model. The target feature cross-correlation information can be input into the target recognition model, which can be a CNN+Attention hybrid architecture model based on deep learning. Then, using the target feature cross-correlation information as the training benchmark, the deviation between the model's output feature prediction results and the target feature cross-correlation information is calculated. The model's network parameters are updated through backpropagation, and then data augmentation algorithms are introduced to simulate feature changes under different data deviations and environmental conditions, optimizing the model's adaptability to multi-working-condition and multi-parameter labeling scenarios. The training process is then repeated until the model's prediction deviation converges to a pre-set threshold, thereby generating the trained target recognition model. This model can accurately identify the positional association between the multi-modal source data to be labeled and the area to be labeled, significantly improving the labeling and positioning accuracy.

[0065] The multimodal data annotation method provided in this application enables the target recognition model to learn more comprehensive and accurate associations of the labeled target features, thereby improving the training effect and positioning accuracy of the target recognition model and ensuring that the generated data annotation information closely matches the actual annotation requirements of the multimodal source data to be annotated, so as to effectively adapt to the multimodal data annotation requirements in different application scenarios.

[0066] Figure 4 The flowchart illustrating the implementation of the multimodal data annotation method provided in Embodiment 4 of this application is shown. The difference between this method and Embodiment 2 is that step S203 specifically includes:

[0067] Step S401: Perform vector transformation based on the multiple target feature information to obtain multiple target feature vectors.

[0068] In this embodiment, the multiple target feature information to be matched are target features extracted from the multimodal source data to be labeled, adapted to different working conditions. These are mostly structured or unstructured feature description data, which cannot be directly input into the target recognition model for training. The system can be based on pre-defined vector transformation rules, which define the standards and processes for converting feature information into vector data. Each target feature information to be matched can then be numerically processed. Alternatively, a Scikit-learn converter or Pandas+NumPy can be used to perform vector transformation, converting different types of feature descriptions such as text, images, and audio into numerical vectors of a uniform dimension. The transformed vector data is then standardized to unify the numerical range of the vectors, eliminating interference from dimensional differences. Finally, each vector data is validated, and invalid vectors are removed, thereby generating multiple target feature vectors to be matched. This ensures that the vector data accurately represents the core attributes of the target feature information to be matched and adapts to the input requirements of the target recognition model.

[0069] Step S402: Integrate the multiple target feature vectors to be matched and the preset multimodal sample feature vectors to generate target feature vectors to be mapped.

[0070] In this embodiment, the preset multimodal sample feature vector can be manually preset. It is a standard feature vector trained based on a large number of similar multimodal labeled samples, which can reflect the common features of similar multimodal data and is used to assist in optimizing the expression of the target feature vector to be matched. First, the multiple target feature vectors to be matched and the preset multimodal sample feature vector can be uniformly processed to ensure that all vectors are in the same dimensional space. Then, a vector concatenation algorithm is used to integrate the multiple target feature vectors to be matched with the preset multimodal sample feature vector, fusing common features and individual features. Then, a feature dimensionality reduction algorithm is used to optimize the integrated vector data, removing redundant dimensions and retaining the core feature dimensions. Finally, the optimized vector data is normalized to ensure the stability of the vector values, thereby generating the target feature vector to be mapped. This vector can comprehensively and accurately represent the target features of the multimodal source data to be labeled.

[0071] Step S403: Generate the feature representation information of the target to be mapped based on the target feature vector to be mapped, the preset target feature query vector to be labeled, the preset target feature key vector to be labeled, and the preset target feature value vector to be labeled.

[0072] In this embodiment, the preset target feature query vector, preset target feature key vector, and preset target feature value vector can all be manually preset and are used for feature query, feature matching, and feature output, respectively. These are the core parameters for generating the target feature representation information. First, the target feature vector to be mapped can be mapped to the preset target feature query vector, preset target feature key vector, and preset target feature value vector to obtain the corresponding query features, key features, and value features. Then, the three types of features are fused to integrate the key information in the features. Next, a feature enhancement algorithm is used to optimize the accuracy of the fused features, highlighting the core features of the target. Finally, the optimized feature information is standardized to ensure the consistency and stability of the feature data, thereby generating the target feature representation information to accurately reflect the feature attributes of the target and provide high-quality feature input for training the target recognition model.

[0073] Step S404: Based on the multiple benchmark labeled target feature information and the target feature representation sequence information to be labeled, train the target identification model to be labeled to obtain the trained target identification model.

[0074] In this embodiment, multiple benchmark labeled target feature information are preset standard labeled target features, serving as benchmark references for model training. The target feature representation sequence information to be labeled is a precise representation of the target features to be labeled. Together, they constitute the input data for training the target recognition model. Multiple benchmark labeled target feature information and the target feature representation sequence information to be labeled can be input into the target recognition model. This model is a CNN+Attention hybrid architecture model based on deep learning. Then, using the multiple benchmark labeled target feature information as standards, the matching error between the target feature representation sequence information to be labeled and the benchmark feature information is calculated. The network parameters of the model are updated through backpropagation. Data augmentation algorithms are then introduced to simulate feature changes under different data biases and environmental conditions, optimizing the model's adaptability and robustness. The training process is repeated until the model matching error converges to a preset threshold, which can be manually preset. This generates a trained target recognition model to accurately match the positional relationship between the multimodal source data to be labeled and the region to be labeled.

[0075] The multimodal data annotation method provided in this application enables the target recognition model to accurately learn the feature patterns of the labeled target, thereby improving the training effect and annotation accuracy of the target recognition model. This effectively solves the problems of poor adaptability and inaccurate positioning of multimodal data annotation under different working conditions, and adapts to the large-scale and high-precision annotation requirements in artificial intelligence model training.

[0076] Figure 5 The flowchart illustrating the implementation of the multimodal data annotation method provided in Embodiment 5 of this application is shown. The difference between this method and Embodiment 4 above is that step S403 specifically includes:

[0077] Step S501: Perform mapping calculations based on the target feature vector to be mapped and the preset target feature query vector to be labeled, and generate target query feature information to be labeled.

[0078] In this embodiment, the target feature vector to be mapped comprehensively represents the target features of the multimodal source data to be labeled. The preset target feature query vector to be labeled can be manually preset and is used to query and extract core features related to the labeled target from the target feature vector to be mapped. It is a key parameter for generating the target query feature information to be labeled. Based on a manually preset mapping calculation rule, the calculation logic between the target feature vector to be mapped and the preset target feature query vector to be labeled is clearly defined. Then, the target feature vector to be mapped and the preset target feature query vector to be labeled are subjected to a dot product operation to calculate their correlation. Based on the correlation, feature components with high matching degree with the query vector in the target feature vector to be mapped are selected. Then, the selected feature components are enhanced to optimize feature accuracy and remove noise interference, thereby generating the target query feature information to be labeled, which is used to accurately extract the core query features in the target feature vector to be mapped.

[0079] Step S502: Perform mapping calculations based on the target feature vector to be mapped and the preset target feature key vector to be labeled to generate target key feature information to be labeled.

[0080] In this embodiment, the preset target feature key vector to be labeled can be manually preset and used to match the target feature vector to be mapped, extracting features that can characterize the key attributes of the labeled target. This feature key vector works in conjunction with the target query feature information to complete the feature weight calculation. The mapping calculation is performed between the target feature vector to be mapped and the preset target feature key vector to be labeled, and then the matching degree is calculated through dot product operation. Key features that highly match the key vector in the target feature vector to be mapped are selected. These selected key features are then standardized to unify the feature dimension and numerical range. Finally, the processed feature information is validated to ensure that the features accurately reflect the key attributes of the labeled target, thereby generating the target key feature information.

[0081] Step S503: Perform mapping calculations based on the target feature vector to be mapped and the preset target feature value vector to be labeled to generate target value feature information to be labeled.

[0082] In this embodiment, the preset target feature value vector to be labeled can be manually preset and is used to extract numerical features from the target feature vector to be mapped that reflect the specific attributes of the labeled target. It is the core foundation for generating the target feature representation vector to be labeled. The mapping calculation rules described above can be followed to perform mapping calculations between the target feature vector to be mapped and the preset target feature value vector to be labeled. Then, the numerical feature components in the target feature vector to be mapped are extracted through vector multiplication. The extracted numerical features are then normalized to ensure the stability and consistency of the feature values. Finally, the processed numerical features are optimized to remove redundant components and retain the core numerical features, thereby generating the target value feature information to be labeled.

[0083] Step S504: Based on the target query feature information and the target key feature information to be labeled, obtain the target feature weight information to be labeled.

[0084] In this embodiment, the query feature information of the target to be labeled is used to clarify the query direction of the target, and the key feature information of the target to be labeled is used to determine the key attributes of the target. The combination of the two can accurately calculate the importance of different features. First, the query feature information and the key feature information of the target to be labeled can be fused to integrate their core features. Then, a weighted calculation algorithm is used to calculate the association weights of the two types of features, quantifying the importance of different features in target identification. Next, the calculated weight data is normalized to ensure that the sum of the weights is 1, conforming to the standard specifications for weight calculation. Finally, the normalized weight data is verified to eliminate abnormal weight values, thereby obtaining the feature weight information of the target to be labeled.

[0085] Step S505: Based on the target feature weight information and the target value feature information to be labeled, obtain the target feature representation vector to be labeled.

[0086] In this embodiment, the target feature weight information clarifies the importance of different features, while the target value feature information characterizes the specific attribute features of the target. Combining the two generates an accurate target feature representation vector. The target value feature information and the target feature weight information can be weighted and fused together, i.e., each value feature is multiplied by its corresponding weight value. Then, all weighted feature values ​​are summed and integrated to obtain a preliminary feature representation vector. This preliminary feature representation vector is then optimized to remove invalid feature components and strengthen the expression of core features. Finally, the optimized vector is standardized to ensure its stability and consistency, thus obtaining the target feature representation vector.

[0087] Step S506: Normalize the target feature representation vector to be labeled to generate target feature representation information.

[0088] In this embodiment, although the feature representation vector of the target to be labeled can reflect the feature attributes of the labeled target, it may have problems such as inconsistent numerical ranges and feature deviations, which need to be optimized through normalization. Min-Max normalization algorithms or Z-Score normalization algorithms can be used to unify the numerical range of the feature vectors, eliminate interference caused by dimensional differences, and then perform normalization operations on the feature representation vector of the target to be labeled, mapping the vector values ​​to a preset reasonable range. The normalized vector data is then verified to ensure the validity and stability of the vector data. Finally, the normalized feature representation vector of the target to be labeled is converted into structured feature information, thereby generating the feature representation information of the target to be labeled.

[0089] The multimodal data annotation method provided in this application ensures that the generated target feature representation information is accurate, comprehensive and standardized, enabling the target recognition model to accurately learn the feature patterns of the labeled target, effectively improving the training accuracy and annotation positioning accuracy of the target recognition model, thereby optimizing the effect and efficiency of multimodal data annotation and meeting the diverse needs for high-precision annotation of multimodal data in artificial intelligence model training.

[0090] Figure 6 The flowchart illustrating the implementation of the multimodal data annotation method provided in Embodiment Six of this application is shown. The difference between this method and Embodiment Five is that step S504 specifically includes:

[0091] Step S601: Multiply the target query feature information and the target key feature information to be labeled to obtain the target query key feature information.

[0092] In this embodiment, the target query feature information is used to extract the core features from the target feature vector to be mapped, and the target key feature information is used to measure the correlation between the target feature vector and other feature vectors. The multiplication of the two can accurately integrate their feature advantages, achieving feature complementarity and enhancement. Specifically, the target query feature information and the target key feature information can be multiplied according to their corresponding dimensions. During the operation, each set of core features and correlation parameters is accurately matched. The operation results can be initially standardized to correct deviations generated during the calculation and remove redundant feature data, thereby generating the target query key feature information, which simultaneously reflects the core features of the target and the correlation between different features.

[0093] Step S602: The target query key feature information and the target query feature information to be labeled are added together to obtain the target query enhancement feature information to be labeled.

[0094] In this embodiment, the query key feature information of the target to be labeled integrates core features and relevance information, while the query feature information of the target to be labeled focuses on the accurate retrieval of core features. Adding the two together further strengthens the proportion of core features while preserving the effectiveness of relevance information. Specifically, the query key feature information and the query feature information of the target to be labeled are added element-wise according to their corresponding feature dimensions. This ensures that core features are further strengthened and relevance information is effectively preserved. The result of the addition is then subjected to feature enhancement processing to optimize feature accuracy, correct minor deviations generated during the addition process, and remove invalid feature data. This generates enhanced query feature information for the target to be labeled, highlighting the core retrieval features of the target while clarifying the relevance weights between different features, thus improving the targeting and effectiveness of the features.

[0095] Step S603: The target query key feature information to be labeled and the target key feature information to be labeled are added together to obtain the target key enhancement feature information to be labeled.

[0096] In this embodiment, the query key feature information of the target to be labeled integrates the dual attributes of query features and key features. The key feature information of the target to be labeled focuses on measuring the degree of feature association. The addition of the two can further strengthen the feature association attribute and improve the expressive strength of the key feature. Specifically, the query key feature information and the key feature information of the target to be labeled can be added element-wise according to the corresponding feature dimensions to strengthen the association expression between features. Then, the result after addition is subjected to stability correction to eliminate numerical fluctuations generated during the calculation process and remove redundant interference features. Finally, the corrected feature information is validated to ensure that the features can accurately reflect the association attribute of the labeled target, thereby generating the key enhanced feature information of the target to be labeled, which is used to enhance the expressive effect of feature association.

[0097] Step S604: Perform multiplication calculation based on the enhanced feature information of the target to be labeled and the enhanced feature information of the target key to be labeled to obtain the feature weight information of the target to be labeled.

[0098] In this embodiment, the enhanced feature information of the target query highlights the core retrieval features, while the enhanced feature information of the target key strengthens the feature association attributes. The multiplication of the two can accurately quantify the importance of different core features and generate feature weights that meet the model training requirements. Specifically, the enhanced feature information of the target query and the enhanced feature information of the target key can be multiplied in the corresponding dimension to further strengthen the matching relationship between core features and association. Then, the calculation result is normalized to ensure that the sum of all feature weights is 1, accurately quantifying the importance ratio of each target feature to be labeled, correcting weight bias, and thus generating target feature weight information to accurately reflect the importance of different target features to be labeled.

[0099] The multimodal data annotation method provided in this application improves the accuracy and relevance of the feature weight information of the target to be annotated, provides a more reliable input basis for the training of the target recognition model, improves the accuracy and anti-interference ability of the target recognition model, and ensures that the generated data annotation information is more in line with the actual annotation needs, thereby improving the consistency and accuracy of multimodal data annotation.

[0100] Figure 7 The flowchart illustrating the implementation of the multimodal data annotation method provided in Embodiment Seven of this application is shown. The difference between this method and Embodiment One is that, after step S104, the method further includes:

[0101] Step S701: Based on the preset mapping relationship between data annotation information and annotation area control information, generate annotation area control information according to the data annotation information.

[0102] In this embodiment, the mapping relationship between the preset data annotation information and the annotation region control information can be pre-set manually. This refers to a standardized mapping rule library built based on a large amount of multimodal annotation experimental data. This library covers the correspondence between parameters such as annotation path, annotation accuracy, and annotation speed in the data annotation information and annotation region control parameters, used to achieve precise adaptation and control of the annotation region. The data annotation information includes core parameters such as the annotation path and annotation range of the region to be annotated. Different annotation paths and accuracy requirements correspond to different region control requirements. For example, when the data annotation information shows a complex multi-region annotation path with high annotation accuracy, high-precision region control parameters need to be generated to ensure that the region deviation is controlled within a preset range. When the annotation path is a simple single region with a fast annotation speed, the region control accuracy can be appropriately adjusted to balance efficiency and accuracy. Specifically, the data annotation information can be input into a preset mapping rule library to retrieve the corresponding regional control parameters. Then, through parameter optimization algorithms, combined with the characteristics of the dimensional information and the range information of the sample to be annotated, the retrieved regional control parameters can be fine-tuned to correct the control deviation, thereby generating annotation region control information. This information can be sent to the annotation region control module of the control annotation terminal in the form of control commands, controlling the annotation terminal to adjust according to the specified regional range and position, achieving precise control of the annotation region, ensuring that the annotation region is completely aligned with the target to be annotated, and avoiding annotation offset caused by regional deviation.

[0103] Step S702: Based on the preset mapping relationship between data annotation information and annotation quality adjustment information, generate annotation quality adjustment information according to the data annotation information; the annotation quality adjustment information includes annotation granularity adjustment information, annotation confidence balance adjustment information, and annotation boundary smoothness adjustment information.

[0104] In this embodiment, the pre-defined mapping relationship between data annotation information and annotation quality adjustment information can be manually preset. It is a multi-dimensional mapping rule constructed based on different sample types and different annotation label requirements. It covers the correspondence between parameters such as annotation accuracy, annotation confidence, and label level in the data annotation information and annotation quality parameters, and is used to achieve accurate matching between annotation quality and annotation requirements. Different parameters in the data annotation information correspond to different quality adjustment requirements. For example, when the data annotation information shows that the label system information to be annotated is a fine-grained multi-level label and the dimensional information of the sample to be annotated is a high-resolution sample, adjustment information with high annotation granularity, high annotation confidence balance, and high annotation boundary smoothness needs to be generated to ensure clear annotation, regular boundaries, and uniform confidence. When the label to be annotated is a coarse-grained single label and the sample resolution is low, adjustment information with moderate annotation granularity and normal confidence balance can be generated to balance annotation efficiency and quality. Specifically, this can be achieved by first analyzing the core parameters in the data annotation information, combining the characteristics of the dimensional information of the samples to be annotated and the label system information to be annotated, and then retrieving a preset mapping relationship library to obtain preliminary annotation quality adjustment parameters. Then, an annotation quality optimization algorithm is used to correct these preliminary parameters, ensuring that the annotation granularity is adapted to the label level, the annotation confidence level is balanced to match the overall annotation consistency, and the annotation boundary smoothness is adapted to the region contour accuracy. This generates annotation quality adjustment information that includes annotation granularity adjustment information, annotation confidence level balance adjustment information, and annotation boundary smoothness adjustment information. This adjustment information is then sent to the annotation quality control module, which controls the annotation granularity by adjusting the sampling density of the annotation terminal, achieves annotation confidence level balance through the confidence level calibration module, and improves annotation boundary smoothness through the boundary optimization module. This results in precise adjustment of annotation quality, thereby improving the accuracy and consistency of multimodal data annotation.

[0105] The multimodal data annotation method provided in this application precisely controls the annotation area and annotation quality, making the multimodal data annotation more suitable for the characteristics of the sample to be annotated and the label requirements, thereby improving the accuracy, stability and annotation quality of multimodal data annotation, so as to meet the stringent requirements of high-precision and large-scale annotation of multimodal data in artificial intelligence model training.

[0106] Corresponding to the method in the above embodiments, Figure 8 A structural block diagram of the multimodal data annotation system provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown. Figure 8 The example multimodal data annotation system can be the execution entity of the multimodal data annotation method provided in the aforementioned embodiment 1.

[0107] Reference Figure 8 The multimodal data annotation system includes:

[0108] The information acquisition module 810 is used to acquire the multimodal source data to be labeled, the dimensional information of the samples to be labeled, the label system information to be labeled, the range information to be labeled, the historical labeling path information, and the target recognition model to be labeled.

[0109] The coordinate information generation module 820 for the region to be labeled is used to obtain the coordinate information of the region to be labeled based on the multimodal source data to be labeled, the dimension information of the sample to be labeled, the range information to be labeled, and the historical labeling path information.

[0110] The current annotation path information generation module 830 is used to generate current annotation path information based on the label system information to be annotated, the annotation range information, and the coordinate information of the area to be annotated;

[0111] The data annotation information generation module 840 is used to generate data annotation information based on the current annotation path information, the label system information to be annotated, the target recognition model to be annotated, and the preset data annotation parameter set.

[0112] For details on how each module in the multimodal data annotation system provided in this application implements its respective function, please refer to the foregoing. Figure 1 The description of Embodiment 1 shown will not be repeated here.

[0113] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0114] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0115] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0116] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0117] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. It should also be understood that although the terms "first," "second," etc., are used in the text to describe various elements in some embodiments of this application, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first table may be named a second table, and similarly, a second table may be named a first table, without departing from the scope of the various described embodiments. Both the first table and the second table are tables, but they are not the same table.

[0118] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0119] The multimodal data annotation method provided in this application can be applied to terminal devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality / virtual reality devices, laptops, super mobile personal computers, netbooks, and personal digital assistants. This application does not impose any restrictions on the specific type of terminal device.

[0120] For example, the terminal device may be a station in a WLAN, a cellular phone, a cordless phone, a session initiation protocol phone, a wireless local loop station, a personal digital processing device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a set-top box, a user premises equipment, and / or other devices for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G networks or mobile terminals in future evolved public terrestrial mobile networks, etc.

[0121] Figure 9 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. For example... Figure 9 As shown, the terminal device 9 of this embodiment includes: at least one processor 90 ( Figure 9(Only one is shown in the image) a memory 91, which stores a computer program 92 that can run on the processor 90. When the processor 90 executes the computer program 92, it implements the steps in the various embodiments of the multimodal data annotation methods described above, for example... Figure 1 Steps S101 to S104 are shown. Alternatively, when the processor 90 executes the computer program 92, it implements the functions of each module / unit in the above system embodiments, for example... Figure 8 The functions of modules 810 to 840 are shown.

[0122] The terminal device 9 can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will understand that... Figure 9 This is merely an example of terminal device 9 and does not constitute a limitation on terminal device 9. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input transmission devices, network access devices, buses, etc.

[0123] The processor 90 may be a central processing unit, or it may be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0124] In some embodiments, the memory 91 may be an internal storage unit of the terminal device 9, such as a hard disk or memory of the terminal device 9. The memory 91 may also be an external storage device of the terminal device 9, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc., equipped on the terminal device 9. Furthermore, the memory 91 may include both internal and external storage units of the terminal device 9. The memory 91 is used to store operating systems, applications, bootloaders, data, and other programs, such as the program code of computer programs. The memory 91 can also be used to temporarily store data that has been sent or will be sent.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] This application also provides a terminal device, which includes at least one memory, at least one processor, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, it causes the terminal device to implement the steps in any of the above method embodiments.

[0127] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0128] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0129] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0130] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0131] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multimodal data annotation method, characterized in that, include: Acquire the multimodal source data to be labeled, the dimensional information of the samples to be labeled, the label system information to be labeled, the label range information, the historical labeling path information, and the target recognition model to be labeled; Based on the multimodal source data to be labeled, the dimensional information of the sample to be labeled, the range information to be labeled, and the historical labeling path information, the coordinate information of the region to be labeled is obtained; Based on the information of the label system to be labeled, the information of the range to be labeled, and the coordinate information of the area to be labeled, the current labeling path information is generated; Data annotation information is generated based on the current annotation path information, the label system information to be annotated, the target recognition model to be annotated, and the preset data annotation parameter set.

2. The multimodal data annotation method as described in claim 1, characterized in that, The step of generating data annotation information based on the current annotation path information, the label system information to be annotated, the target recognition model to be annotated, and the preset data annotation parameter set specifically includes: Based on the preset feature extraction rules for the target to be labeled, and according to the preset set of data labeling parameters, multiple benchmark labeled target feature information are generated; Feature extraction processing is performed on the unlabeled multimodal source data to obtain multiple target feature information to be matched; Based on the multiple benchmark labeled target feature information and multiple target feature information to be matched, the target recognition model to be labeled is trained to obtain the trained target recognition model to be labeled. Based on the current annotation path information, the label system information to be annotated, and the trained target recognition model to be annotated, data annotation information is generated.

3. The multimodal data annotation method as described in claim 2, characterized in that, The multiple benchmark annotation target feature information includes first benchmark annotation target feature information and second benchmark annotation target feature information; The plurality of target feature information to be matched includes first target feature information to be matched and second target feature information to be matched; The step of training the target recognition model based on the multiple benchmark labeled target feature information and the multiple target feature information to be matched, to obtain the trained target recognition model, specifically includes: The first target feature fusion information is generated by fusing the first benchmark labeled target feature information and the first target feature information to be matched. The second target feature fusion information is generated by fusing the second benchmark labeled target feature information and the second target feature information to be matched. Based on the first target feature fusion information and the second target feature fusion information, correlation processing is performed to generate target feature cross-correlation information; Based on the cross-correlation information of the target features, the target recognition model to be labeled is trained to obtain the trained target recognition model.

4. The multimodal data annotation method as described in claim 2, characterized in that, The step of training the target recognition model based on the multiple benchmark labeled target feature information and the multiple target feature information to be matched, to obtain the trained target recognition model, specifically includes: Based on the feature information of the multiple targets to be matched, vector transformation is performed to obtain multiple feature vectors of the targets to be matched; The multiple target feature vectors to be matched and the preset multimodal sample feature vectors are integrated and processed to generate the target feature vector to be mapped. Based on the target feature vector to be mapped, the preset target feature query vector to be labeled, the preset target feature key vector to be labeled, and the preset target feature value vector to be labeled, the target feature representation information to be labeled is generated. Based on the target feature information of the multiple benchmark labels and the feature representation sequence information of the target to be labeled, the target recognition model to be labeled is trained to obtain the trained target recognition model.

5. The multimodal data annotation method as described in claim 4, characterized in that, The step of generating the target feature representation information to be labeled based on the target feature vector to be mapped, the preset target feature query vector to be labeled, the preset target feature key vector to be labeled, and the preset target feature value vector to be labeled specifically includes: Mapping calculations are performed based on the target feature vector to be mapped and the preset target feature query vector to be labeled to generate target query feature information to be labeled. Mapping calculations are performed based on the target feature vector to be mapped and the preset target feature key vector to be labeled to generate target key feature information to be labeled. Mapping calculations are performed based on the target feature vector to be mapped and the preset target feature value vector to be labeled to generate target value feature information to be labeled. Based on the target query feature information and target key feature information to be labeled, the target feature weight information is obtained; Based on the target feature weight information and the target value feature information to be labeled, the target feature representation vector to be labeled is obtained; The feature representation vector of the target to be labeled is normalized to generate the feature representation information of the target to be labeled.

6. The multimodal data annotation method as described in claim 5, characterized in that, The step of obtaining the feature weight information of the target to be labeled based on the query feature information and key feature information of the target to be labeled specifically includes: The target query key feature information to be labeled is obtained by multiplying the target query feature information and the target key feature information to be labeled. The target query key feature information and the target query feature information to be labeled are added together to obtain the target query enhancement feature information; The enhanced feature information of the target key is obtained by adding the query key feature information and the target key feature information to be labeled. The target feature weight information is obtained by multiplying the target query enhancement feature information and the target key enhancement feature information.

7. The multimodal data annotation method as described in claim 1, characterized in that, After the step of generating data annotation information based on the current annotation path information, the label system information to be annotated, the target recognition model to be annotated, and the preset data annotation parameter set, the method further includes: Based on the preset mapping relationship between data annotation information and annotation area control information, annotation area control information is generated according to the data annotation information; Based on the preset mapping relationship between data annotation information and annotation quality adjustment information, annotation quality adjustment information is generated according to the data annotation information; the annotation quality adjustment information includes annotation granularity adjustment information, annotation confidence balance adjustment information, and annotation boundary smoothness adjustment information.

8. A multimodal data annotation system, characterized in that, include: The information acquisition module is used to acquire the multimodal source data to be labeled, the dimensional information of the samples to be labeled, the label system information to be labeled, the range information to be labeled, the historical labeling path information, and the target recognition model to be labeled. The module for generating coordinate information of the region to be labeled is used to obtain the coordinate information of the region to be labeled based on the multimodal source data to be labeled, the dimensional information of the sample to be labeled, the range information to be labeled, and the historical labeling path information. The current annotation path information generation module is used to generate current annotation path information based on the label system information to be annotated, the annotation range information, and the coordinate information of the area to be annotated; The data annotation information generation module is used to generate data annotation information based on the current annotation path information, the label system information to be annotated, the target recognition model to be annotated, and the preset data annotation parameter set.

9. A terminal device, characterized in that, The terminal device includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images