Annotation method and device based on multi-modal model

Abstract

The embodiment of the application provides a kind of based on the labeling method and device of multimodal model, method includes: by the customer terminal dataset is input to preset initial model, according to the text data segmented by setting adaptive word segmentation algorithm, obtain text vector, according to preset image segmentation algorithm, image data is segmented, respectively to the target image region, foreground image region and background image region obtained after segmentation are carried out feature extraction, obtain image vector, according to preset multi-scale feature extraction network, the multi-scale point cloud feature of three-dimensional point cloud data is extracted, obtain point cloud vector, text vector, image vector and point cloud vector are input to the main network of preset initial model and carry out matching space construction operation, determine corresponding multimodal model, according to multimodal model, customer terminal data is predicted and labeled operation, determine the prediction result of customer terminal data output by multimodal model, the efficiency and accuracy of data labeling can be improved by the application.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a labeling method and apparatus based on a multimodal model. Background Technology

[0002] In the field of data annotation, with the rapid development of big data and artificial intelligence technologies, the requirements for the efficiency and accuracy of data annotation are increasing. Traditional data annotation methods rely on manual annotation, which is inefficient and costly. To improve annotation efficiency, AI-based automatic annotation methods have emerged.

[0003] Existing AI-based methods for improving annotation efficiency primarily involve designing corresponding AI models based on the application scenarios proposed by clients and training these models using existing data from service providers. The trained AI models are then used for annotation work to improve efficiency. However, this approach suffers from several major problems:

[0004] 1. High degree of model customization: Different clients have very different needs, requiring the design of different models and training on different data. This places high demands on the service provider's infrastructure, algorithm development capabilities, and data reserves.

[0005] 2. Difficulty in addressing cross-domain data business needs: If the data in a customer's business needs is cross-domain with the supplier's data (e.g., different sensors, different countries, cities, or seasons for data collection), the performance of models trained on existing data will be severely degraded on the customer's data, and AI capabilities will be unable to significantly improve annotation efficiency.

[0006] Based on the above problems, existing AI-based annotation methods cannot meet the efficiency and accuracy requirements of the data annotation field. Summary of the Invention

[0007] To address the problems in the existing technology, this application provides a labeling method and apparatus based on a multimodal model, which can improve the efficiency and accuracy of data labeling.

[0008] To solve at least one of the above problems, this application provides the following technical solution:

[0009] Firstly, this application provides an annotation method based on a multimodal model, including:

[0010] Receive a client terminal dataset, which includes text data, image data, and 3D point cloud data;

[0011] The client terminal dataset is input into the data preprocessing module of a preset initial model. The text data is segmented using a predefined adaptive word segmentation algorithm to determine the corresponding text vectors. The image data is segmented using a predefined image segmentation algorithm to determine the corresponding target image region, foreground image region, and background image region. High-resolution feature extraction is performed on the foreground image region to determine the corresponding foreground features. Blur feature extraction is performed on the background image region to determine the corresponding background features. Feature extraction is performed on the target image region to determine the corresponding target features. The target features, foreground features, and background features are vectorized to determine the corresponding image vectors. Multi-scale point cloud feature extraction is performed on the 3D point cloud data using a predefined multi-scale feature extraction network to determine the corresponding point cloud vectors. The text vectors, image vectors, and point cloud vectors are input into the backbone network of the preset initial model for matching space construction to determine the corresponding multimodal model.

[0012] The client terminal data is predicted and labeled according to the multimodal model to determine the prediction result of the client terminal data output by the multimodal model, wherein the prediction result includes predicted text annotation, predicted image annotation and predicted 3D point cloud annotation.

[0013] Further, before segmenting the text data according to the set word segmentation algorithm and determining the corresponding text vector, the following steps are included:

[0014] Collect historical text datasets, perform word segmentation on the historical text datasets, and determine the corresponding smallest text subunits;

[0015] The smallest text subunit is input into a preset long short-term network for model training to determine the dependency relationship between the smallest text subunits output by the long short-term network.

[0016] Based on the dependencies, the preset validation set dependencies, and the preset cross-entropy loss function, the corresponding loss function is determined, and the parameters of the long short-term network are updated according to the optimal loss function and the reverse update algorithm to determine the corresponding adaptive word segmentation algorithm.

[0017] Further, the step of segmenting the text data according to the set word segmentation algorithm to determine the corresponding text vector includes:

[0018] The text data is analyzed according to the set word segmentation algorithm to determine the corresponding text data segmentation points. The text data is then segmented according to the text data segmentation points to determine the corresponding optimal text subunits.

[0019] The optimal text subunit is vectorized according to a preset word vector model to determine the corresponding text vector.

[0020] Further, the step of performing multi-scale point cloud feature extraction on the 3D point cloud data according to a preset multi-scale feature extraction network to determine the corresponding point cloud vector includes:

[0021] The three-dimensional point cloud data is sampled according to a preset far-point sampling algorithm to determine the corresponding centroid point. The neighborhood construction operation is performed on the centroid point according to a preset multi-scale radius to determine the corresponding multi-scale neighborhood.

[0022] Feature extraction is performed on the multi-scale neighborhood, and feature fusion is performed on the multi-scale neighborhood features obtained after the feature extraction to determine the multi-scale features corresponding to the centroid point. The multi-scale features are then vectorized to determine the corresponding point cloud vector.

[0023] Further, the step of inputting the text vector, the image vector, and the point cloud vector into the backbone network of a preset initial model to perform a matching space construction operation and determine the corresponding multimodal model includes:

[0024] The text vector, the image vector, and the point cloud vector are input into the backbone network of a preset initial model. Distance measurement is performed on the text vector, the image vector, and the point cloud vector according to a preset shared space to determine the corresponding alignment.

[0025] Based on the alignment, a matching space construction operation is performed on the text vector, the image vector, and the point cloud vector to determine the corresponding multimodal vector mapping relationship, and the corresponding multimodal model is determined based on the multimodal vector mapping relationship.

[0026] Further, after performing prediction and annotation operations on the customer terminal data based on the multimodal model and determining the prediction result of the customer terminal data output by the multimodal model, the process includes:

[0027] A difference comparison operation is performed between the predicted result and the set actual result to determine the corresponding deviation value.

[0028] The multimodal model is updated based on the deviation value and the backpropagation algorithm to determine the updated multimodal model, wherein the set true result is determined by the annotator.

[0029] Further, before performing the difference comparison operation based on the predicted result and the set true result to determine the corresponding deviation value, the following steps are included:

[0030] The annotator manually annotates the dynamic data in the client terminal data on the annotator's terminal to determine the corresponding dynamic data enveloping frame;

[0031] The annotator manually annotates the static data in the client terminal data on the annotator's terminal to determine the corresponding static data polyline.

[0032] The corresponding true result is determined based on the dynamic data enveloping frame and the static data polyline.

[0033] Secondly, this application provides a labeling device based on a multimodal model, comprising:

[0034] The data receiving module is used to receive client terminal datasets, which include text data, image data, and 3D point cloud data.

[0035] The multimodal model training module is used to input the client terminal dataset into the data preprocessing module of a preset initial model, perform segmentation operations on the text data according to a set adaptive word segmentation algorithm to determine the corresponding text vectors, perform segmentation operations on the image data according to a preset image segmentation algorithm to determine the corresponding target image region, foreground image region, and background image region, perform high-resolution feature extraction operations on the foreground image region to determine the corresponding foreground features, perform blurred feature extraction operations on the background image region to determine the corresponding background features, perform feature extraction operations on the target image region to determine the corresponding target features, perform vectorization operations on the target features, the foreground features, and the background features to determine the corresponding image vectors, perform multi-scale point cloud feature extraction operations on the three-dimensional point cloud data according to a preset multi-scale feature extraction network to determine the corresponding point cloud vectors, and input the text vectors, the image vectors, and the point cloud vectors into the backbone network of the preset initial model to perform matching space construction operations to determine the corresponding multimodal model;

[0036] The prediction annotation module is used to perform prediction annotation operations on the client terminal data according to the multimodal model, and determine the prediction results of the client terminal data output by the multimodal model, wherein the prediction results include predicted text annotations, predicted image annotations, and predicted 3D point cloud annotations.

[0037] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the annotation method based on the multimodal model.

[0038] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the annotation method based on the multimodal model.

[0039] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the annotation method based on the multimodal model.

[0040] As can be seen from the above technical solution, this application provides a labeling method and apparatus based on a multimodal model. The method involves inputting a client terminal dataset into a preset initial model, segmenting text data using a set adaptive word segmentation algorithm to obtain text vectors, segmenting image data using a preset image segmentation algorithm, extracting features from the segmented target image region, foreground image region, and background image region to obtain image vectors, extracting multi-scale point cloud features from the 3D point cloud data using a preset multi-scale feature extraction network to obtain point cloud vectors, inputting the text vectors, image vectors, and point cloud vectors into the backbone network of the preset initial model for matching space construction to determine the corresponding multimodal model, and performing prediction labeling operations on the client terminal data based on the multimodal model to determine the prediction results of the client terminal data output by the multimodal model. This improves the efficiency and accuracy of data labeling. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is one of the flowcharts illustrating the annotation method based on a multimodal model in the embodiments of this application;

[0043] Figure 2 This is the second flowchart illustrating the annotation method based on a multimodal model in the embodiments of this application;

[0044] Figure 3 This is the third flowchart illustrating the annotation method based on a multimodal model in the embodiments of this application;

[0045] Figure 4 This is the fourth flowchart illustrating the annotation method based on a multimodal model in the embodiments of this application;

[0046] Figure 5 This is the fifth flowchart illustrating the annotation method based on a multimodal model in the embodiments of this application;

[0047] Figure 6 This is the sixth flowchart illustrating the annotation method based on a multimodal model in the embodiments of this application;

[0048] Figure 7 This is the seventh flowchart illustrating the annotation method based on a multimodal model in the embodiments of this application;

[0049] Figure 8 This is a structural diagram of the annotation device based on a multimodal model in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0051] Figure label:

[0052] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0055] Considering that existing AI-based annotation methods cannot meet the efficiency and accuracy requirements of data annotation in the field, this application provides an annotation method and apparatus based on a multimodal model. The method involves inputting a client terminal dataset into a preset initial model, segmenting text data using a predefined adaptive word segmentation algorithm to obtain text vectors, segmenting image data using a preset image segmentation algorithm, extracting features from the segmented target image region, foreground image region, and background image region to obtain image vectors, extracting multi-scale point cloud features from 3D point cloud data using a preset multi-scale feature extraction network to obtain point cloud vectors, inputting the text vectors, image vectors, and point cloud vectors into the backbone network of the preset initial model for matching space construction to determine the corresponding multimodal model, and performing prediction annotation operations on the client terminal data based on the multimodal model to determine the prediction results of the client terminal data output by the multimodal model. This improves the efficiency and accuracy of data annotation.

[0056] To improve the efficiency and accuracy of data annotation, this application provides an embodiment of an annotation method based on a multimodal model, see [link to embodiment]. Figure 1 The annotation method based on the multimodal model specifically includes the following:

[0057] Step S101: Receive client terminal dataset, which includes text data, image data, and 3D point cloud data;

[0058] Optionally, in this embodiment, the client terminal dataset is used to train the multimodal model. Therefore, the dataset needs to have multiple data formats, and the labeled data includes text, images (2D data), 3D point cloud data, etc.

[0059] Understandably, compared to existing technologies, traditional models often only process specific types of single data (such as images or text), lacking a comprehensive understanding of different data modalities. Therefore, when dealing with complex and diverse annotation tasks, such as images containing text, traditional models have limited performance. This technical solution, through multimodal data training, enables the model to better adapt to various data sources and application scenarios.

[0060] Step S102: Input the client terminal dataset into the data preprocessing module of the preset initial model, perform segmentation operation on the text data according to the set adaptive word segmentation algorithm to determine the corresponding text vector, perform segmentation operation on the image data according to the preset image segmentation algorithm to determine the corresponding target image region, foreground image region and background image region, perform high-definition feature extraction operation on the foreground image region to determine the corresponding foreground features, perform fuzzy feature extraction operation on the background image region to determine the corresponding background features, perform feature extraction operation on the target image region to determine the corresponding target features, perform vectorization operation on the target features, the foreground features and the background features to determine the corresponding image vector, perform multi-scale point cloud feature extraction operation on the three-dimensional point cloud data according to the preset multi-scale feature extraction network to determine the corresponding point cloud vector, input the text vector, the image vector and the point cloud vector into the backbone network of the preset initial model to perform matching space construction operation to determine the corresponding multimodal model;

[0061] Optionally, in this embodiment, this step innovates in the data preprocessing section by converting different types of data (text, images, point clouds, etc.) into a unified token (unit element) format and sending it into the Transformer (deep learning model) architecture for processing. This enables data from different modalities to perform interactive reasoning in the same model, increasing the model's flexibility and adaptability.

[0062] Optionally, in this embodiment, the data collected by the terminal enters the preprocessing module of the Multimodal Large Model (MMLM) for preprocessing. Different types of data (such as images, text, and point clouds) are converted into tokens of a unified format through different preprocessing processes. For example, text is tokenized, while image and point cloud data are converted into tokens through patch segmentation and feature extraction.

[0063] Optionally, in this embodiment, the preprocessed Token is input into the inference module of the Multimodal Large Model (MMLM). Through the matching space, the AI ​​model will output the inference result. In the subsequent step S103, the model is trained through the feedback mechanism.

[0064] Optionally, in this embodiment, in the multimodal AI model (MMLM model based on the Transformer architecture), the preprocessing step is a crucial stage in transforming the input data into a form that the model can process. For data of different modalities, the core task of preprocessing is tokenization to ensure that various types of data can be uniformly represented and input into the neural network for processing.

[0065] Optionally, in this embodiment, the text data is tokenized.

[0066] Optionally, during the tokenization process, Natural Language Processing (NLP) techniques are used, such as WordPiece (word segmentation algorithm), Byte Pair Encoding (BPE) (machine translation algorithm), or SentencePiece (sub-word matching algorithm). These methods decompose the input text into words or sub-word units, which are then converted into corresponding tokens. The main purpose of tokenization is to transform the text content into a series of digital tokens, which are then fed into the model as an input sequence.

[0067] Preferably, for long or compound words in the text, an adaptive word segmentation algorithm is used to divide the vocabulary into more appropriate subwords. This method can better handle polysemous and rare words, and improve the model's adaptability to different languages.

[0068] Specifically, adaptive word segmentation algorithms use contextual models like LSTM to process text, extracting the semantic information of each word or character within its context, and then dynamically adjusting the segmentation points based on this information. When processing sequential data, LSTM can remember information from preceding and following sequences, thus determining whether the current word needs further segmentation by analyzing the context of the preceding and following words. For example, when a word is a compound word in a specific context, LSTM can adjust its segmentation strategy through its memory mechanism.

[0069] Another implementation involves adaptive word segmentation algorithms that use contextual models like the Transformer to process text. Through a self-attention mechanism, the Transformer can simultaneously consider the contextual information of the entire sentence at every position in the text. This self-attention mechanism allows the model to adjust the segmentation positions based on the global context. After capturing the relationship between each word in the context, the model dynamically adjusts the segmentation boundaries based on the contextual information.

[0070] It's worth noting that the text data first undergoes word segmentation, after which it is converted into tokens and input into the model for processing. At this stage, the model adjusts word boundaries through contextual analysis to ensure more accurate text processing. During joint training, preprocessed text, images, point clouds, and other data from different modalities are fused within the model. This joint training enables the model to understand the relationships between different data types. Throughout this process, the word segmentation is also optimized based on its context within the entire multimodal task.

[0071] Finally, the text processed by adaptive word segmentation is transformed into vector patterns through word embedding, laying the foundation for the subsequent construction of the matching space using a unified token input model backbone network.

[0072] Optionally, in this embodiment, the image data is tokenized.

[0073] Optionally, during image data preprocessing, different processing strategies can be applied based on different regions of the image (foreground, background, target region, etc.).

[0074] Specifically, the image is first divided into different regions, including foreground, background, and target regions, using image segmentation algorithms (such as U-Net, FCN, etc.). Then, based on the different characteristics of these regions, appropriate preprocessing methods are dynamically selected, including contrast enhancement and noise removal for the target region, and specific processing (such as retouching, blurring, and noise reduction) for the background region.

[0075] For example, higher resolution is used in the target and foreground regions to preserve more detail, while lower resolution is used in the background regions to reduce computation. Information from different regions of the image is extracted as feature vectors, and region fitting is performed in the model based on these features. This means the model can identify and process different regions of the image differently. For example, for the foreground region, the model might choose a more detailed feature representation, while for the background region, it might choose a simpler feature representation.

[0076] Understandably, region-based dynamic adjustment methods can flexibly adjust preprocessing methods according to the characteristics of different regions in an image, thereby improving the model's processing efficiency and accuracy. In subsequent model applications, the model can adjust its processing methods for different regions in real time. For example, when an annotator labels a new region, the model can learn and adapt to the characteristics of this new region, dynamically optimizing the processing method.

[0077] Finally, the preprocessed image data is transformed into a unified vector format, laying the foundation for constructing the matching space by inputting a unified token into the model backbone network.

[0078] Optionally, in this embodiment, the 3D point cloud data is tokenized.

[0079] Optionally, point cloud data often contains multi-scale information. In order to process multi-scale point cloud data, it is first necessary to decompose the point cloud data through a hierarchical structure.

[0080] Specifically, in point cloud data processing, a set of key points is first selected from the input point cloud using the Farthest Point Sampling (FPS) strategy. Then, for each point, it not only considers the surrounding local point cloud data but also extracts features through different sampling scales. For example, it can start with a small neighborhood to extract detailed information and then expand to a larger neighborhood to capture more contextual information. Then, from multiple levels, each layer uses local point cloud feature extraction methods to effectively extract details from the point cloud data. Through feature aggregation at different scales, the network can comprehensively consider information from multiple levels, from local details to global structure.

[0081] In terms of performance, the multi-scale feature extraction capability significantly improves the model's adaptability in complex point cloud scenarios. This includes:

[0082] 1) Capturing diverse object forms: Different objects sometimes exhibit significant differences in scale and density in point clouds. PointNet++ can handle these differences and capture various representations of objects through a hierarchical structure.

[0083] 2) Enhanced recognition of small objects: For small objects, the network can focus on details through a fine local feature extraction module, without being disturbed by a large background.

[0084] Finally, the preprocessed 3D point cloud data is converted into a unified vector format, laying the foundation for constructing the matching space by inputting a unified token into the model backbone network.

[0085] Optionally, in this embodiment, the construction of the matching space is a crucial step in the training of a multimodal large model. Its main purpose is to ensure that the model can perform multimodal inference efficiently and achieve fast convergence by effectively matching the features of different modal data (such as text, images, 3D point clouds, etc.).

[0086] Optionally, a neural network based on the Transformer architecture was used, which is capable of handling different types of data.

[0087] Text data: The text is processed through tokenization to generate a series of tokens, which are then input into the Transformer model.

[0088] Image data: The image is divided into multiple patches, and each patch is converted into a vector representation (token) through feature extraction methods.

[0089] 3D point cloud data: The coordinates (e.g., x, y, z) of each point cloud data are processed as tokens, and spatial features are extracted through a specific network structure.

[0090] Optionally, the next step is to calculate the similarity between different modalities to establish a mapping relationship.

[0091] Specifically, by calculating the alignment between different modalities, different modalities (such as images and text, or 3D point clouds and text) are transformed into a shared space. This shared space effectively unifies data from different modalities within the same feature space for comparison and matching. Cross-modal embedding techniques embed data from different modalities into a common feature space. This allows the model to understand the relationships and dependencies between different modalities. Within this shared space, distance metrics (such as Euclidean distance or cosine similarity) are used to calculate the matching degree between different modalities. Two modalities are considered matched only when the distance between their features in the shared space is sufficiently small. Then, for the matched data from different modalities, a self-attention mechanism is used to weightedly fuse multiple modalities such as images, text, and point clouds, enabling the model to dynamically adjust the contribution of data based on the importance of different modalities. This matching space enhances the model's adaptability to complex datasets, improving not only model accuracy but also providing strong support for the effective integration of multimodal learning frameworks.

[0092] Understandably, the matching space is not fixed due to the varying needs of different customers and application scenarios. It is dynamically adjusted based on new labeled data to ensure that each terminal's labeling task receives appropriate training and inference support. Each time a labeling result fails the annotation review, a new training cycle is triggered, updating the model and optimizing the matching space.

[0093] After the matching space is constructed, a multimodal model is obtained, which is trained and optimized based on a feedback mechanism.

[0094] Step S103: Perform prediction and annotation operations on the client terminal data according to the multimodal model, and determine the prediction results of the client terminal data output by the multimodal model, wherein the prediction results include predicted text annotations, predicted image annotations, and predicted 3D point cloud annotations.

[0095] Optionally, this step involves constructing a multimodal model of the matching space to predict and label the client terminal data, obtaining the labeling results. The intelligent agent platform is deployed on the labeler's end, and the labeler's end and the multimodal model perform labeling simultaneously.

[0096] Subsequently, the annotators will evaluate the predictions of the multimodal model. The feedback from the annotators will be compared with the inference results of the large multimodal model. The differences will be used for feedback model iteration and training to achieve rapid convergence of the neural network model and training of the large multimodal model.

[0097] This example demonstrates how this embodiment trains the model to process multimodal data and performs model self-training based on a feedback mechanism to achieve an intelligent annotation paradigm for terminal agents.

[0098] As described above, the annotation method based on a multimodal model provided in this application can improve the efficiency and accuracy of data annotation by inputting a client terminal dataset into a preset initial model, segmenting text data according to a set adaptive word segmentation algorithm to obtain text vectors, segmenting image data according to a preset image segmentation algorithm, extracting features from the segmented target image region, foreground image region, and background image region to obtain image vectors, extracting multi-scale point cloud features from 3D point cloud data according to a preset multi-scale feature extraction network to obtain point cloud vectors, inputting the text vectors, image vectors, and point cloud vectors into the backbone network of the preset initial model for matching space construction to determine the corresponding multimodal model, performing prediction annotation operations on the client terminal data according to the multimodal model, and determining the prediction results of the client terminal data output by the multimodal model.

[0099] In one embodiment of the annotation method based on multimodal models in this application, see [link to documentation]. Figure 2 It can also specifically include the following:

[0100] Step S201: Collect historical text datasets, perform word segmentation on the historical text datasets, and determine the corresponding smallest text subunits;

[0101] Step S202: Input the smallest text subunit into a preset long short-term network for model training, and determine the dependency relationship between the smallest text subunits output by the long short-term network;

[0102] Step S203: Based on the dependencies, the preset validation set dependencies, and the preset cross-entropy loss function, determine the corresponding loss function, and update the parameters of the long short-term network according to the optimal loss function and the reverse update algorithm to determine the corresponding adaptive word segmentation algorithm.

[0103] Optionally, in this embodiment, the purpose of this step is to train an adaptive word segmentation model, which is then used to implement the data preprocessing process in the multimodal model. The core objective of the adaptive word segmentation strategy is to select the most suitable word segmentation method based on the different characteristics of the text data (such as register, content, context, etc.).

[0104] Optionally, in this embodiment, a large amount of historical text data is first collected, and basic text cleaning operations are performed, including removing useless symbols, standardizing punctuation marks, and processing blank characters. For multilingual text data, language recognition is required to classify texts in different languages and select appropriate tokenizers for processing. Stop words (such as "de", "shi", "zai", etc.) are removed to reduce noise and improve the tokenization effect.

[0105] Secondly, a tokenization strategy is selected, and tokenization annotation and processing are performed according to the tokenization strategy.

[0106] Rule-based tokenization: For some structured texts (such as legal documents, technical documents, etc.), rule-driven tokenization methods can be used to perform tokenization based on dictionaries, regular expressions, etc.

[0107] Subword Tokenization: The BPE algorithm is used to split out-of-vocabulary words, breaking words into smaller units to improve the robustness of the model.

[0108] Context-aware tokenization: In some scenarios with rich context information, pre-trained language models (such as BERT) can be used to determine word boundaries. For example, use context information to handle the segmentation of some polysemous words or compound words.

[0109] After tokenization annotation of words, they are input into a long short-term network for model training, so that the long short-term network can recognize the dependencies between words and output the text tokenization predicted by the model according to the dependencies.

[0110] Finally, the cross-entropy loss is calculated between the true value results in the validation set and the text tokenization results predicted by the model. The calculated loss function dynamically adjusts the parameters of the long short-term network through backpropagation until the optimal parameters are obtained, so that the long short-term network can extract the semantic information of each word or character in the text in the context, and then dynamically adjust the segmentation points of the tokenization based on this information.

[0111] Through step S203, this embodiment successfully trains an adaptive tokenization algorithm, laying a foundation for the subsequent text preprocessing process of the multimodal model.

[0112] In an embodiment of the annotation method based on a multimodal model of the present application, referring to Figure 3 , it may specifically include the following content:

[0113] Step S301: Analyze the text data according to the set tokenization algorithm to determine the corresponding text data segmentation points, and perform segmentation operations on the text data according to the text data segmentation points to determine the corresponding optimal text sub-units;

[0114] Step S302: Perform vectorization operation on the optimal text subunit according to the preset word vector model to determine the corresponding text vector.

[0115] Optionally, in this embodiment, the purpose of this step is to preprocess the text data to obtain text vectors, laying the foundation for subsequent joint learning and inference with image vectors and point cloud vectors in the same feature space.

[0116] Optionally, in this embodiment, the text data in the obtained customer terminal data is input into the data preprocessing module, and the text data is segmented according to the adaptive word segmentation algorithm obtained in step S203. Specifically, the adaptive word segmentation algorithm will identify the best segmentation point of the text data and perform word segmentation accordingly.

[0117] For example, suppose the input text is "The annotator manually annotated the text data". After tokenization, it may be decomposed into the following tokens: "["annotator", "manual", "annotation", "text", "data"]".

[0118] In terms of performance, adaptive word segmentation dynamically adjusts the segmentation method according to the needs of different fields. For example, when processing technical documents, it can identify and segment technical terms, while when processing social media text, it can identify and correctly segment some slang, abbreviations, etc.

[0119] Adaptive word segmentation not only relies on dictionaries but also incorporates contextual information, avoiding segmentation errors caused by polysemous words or contextual dependencies, thus improving accuracy. For example, for the word "apple," the segmentation result can automatically adapt to whether it refers to "fruit" or "brand" in the context.

[0120] Adaptive word segmentation effectively handles out-of-vocabulary (OOV) words by segmenting sub-words (such as BPE, WordPiece, etc.). Even when encountering new words, the algorithm can generate appropriate segmentation results based on existing vocabulary and context.

[0121] Finally, through word embedding, these tokens obtained from adaptive word segmentation are further converted into numerical representations that the model can process, such as the corresponding word embeddings.

[0122] Through step S302, this embodiment realizes the segmentation of text data in client terminal data using an adaptive word segmentation algorithm, laying a solid foundation for subsequent joint training of the multimodal model.

[0123] In one embodiment of the annotation method based on multimodal models in this application, see [link to documentation]. Figure 4 It can also specifically include the following:

[0124] Step S401: Sampling operation is performed on the three-dimensional point cloud data according to the preset far-point sampling algorithm to determine the corresponding centroid point. Neighborhood construction operation is performed on the centroid point according to the preset multi-scale radius to determine the corresponding multi-scale neighborhood.

[0125] Step S402: Perform feature extraction on the multi-scale neighborhood, and perform feature fusion on the multi-scale neighborhood features obtained after the feature extraction to determine the multi-scale features corresponding to the centroid point. Perform vectorization on the multi-scale features to determine the corresponding point cloud vector.

[0126] Optionally, in this embodiment, the purpose of this step is to preprocess the three-dimensional point cloud data of the client terminal data to obtain three-dimensional point cloud vectors, laying the foundation for subsequent joint learning and inference with image vectors and text vectors in the same feature space.

[0127] Optionally, in this embodiment, a set of keypoints is first selected from the input point cloud using the Farthest Point Sampling (FPS) strategy. For each keypoint, its neighborhood is determined based on multiple different radii (i.e., different scales). For example, for a center point, neighborhoods with radii of 1 unit, 2 units, and 3 units may be considered simultaneously. For each neighborhood defined by a different radius, local features are extracted using a PointNet structure. Finally, the features extracted from different neighborhoods are concatenated to form a multi-scale feature representation of the center point.

[0128] Optionally, in this embodiment, multiple such feature extraction layers are used. In each layer of the network, features of local regions are extracted, and then multiple local features are aggregated at a higher level to capture information at different scales.

[0129] Finally, the extracted point cloud data is converted into point cloud vectors, laying a solid foundation for subsequent joint training of the multimodal model.

[0130] Through step S402, this embodiment successfully obtained high-precision 3D point cloud features through multi-scale feature extraction, laying the foundation for subsequent training of multimodal models.

[0131] In one embodiment of the annotation method based on multimodal models in this application, see [link to documentation]. Figure 5 It can also specifically include the following:

[0132] Step S501: Input the text vector, the image vector, and the point cloud vector into the backbone network of the preset initial model, and perform distance measurement operation on the text vector, the image vector, and the point cloud vector according to the preset shared space to determine the corresponding alignment.

[0133] Step S502: Perform a matching space construction operation on the text vector, the image vector, and the point cloud vector according to the alignment degree to determine the corresponding multimodal vector mapping relationship, and determine the corresponding multimodal model according to the multimodal vector mapping relationship.

[0134] Optionally, in this embodiment, the purpose of this step is to construct a matching space by converting different types of data (text, images, point clouds, etc.) into a unified token format and sending them into the Transformer architecture for processing in the shared space, so that the trained model has a strong multimodal data processing capability.

[0135] Optionally, in this embodiment, the key to the multimodal AI model lies in unifying data from different modalities (such as text, images, 3D point clouds, etc.) into a token format that the model can process. Through tokenization, all data (whether text, images, or point cloud data) is converted into a standardized token sequence, ensuring that different data sources can interact and perform joint inference within the same model.

[0136] After unified tokenization, text, image, and point cloud data are all converted into tokens and input into the model in a specific order or structure. For example, text token sequences, image patch token sequences, and point cloud coordinate token sequences are combined and input into the input layer of a Transformer model. In terms of effectiveness, tokenization not only transforms data into a unified form but also extracts latent features from the data (e.g., semantics in text, local features in images, spatial relationships in point cloud data), which are then used by the model for inference and learning. After tokenization, all modal data can be processed through the same mechanism in the neural network, ensuring that the model's inference capabilities are effectively utilized across different types of data.

[0137] Optionally, in this embodiment, an effective matching space is constructed, and mapping relationships are established by calculating the similarity between different modal data. That is, through cross-modal embedding techniques, different modalities (such as images and text, or 3D point clouds and text) are transformed into a shared space to calculate their alignment, allowing the model to understand the relationships and interdependencies between different modalities. The shared space effectively unifies different modal data within the same feature space for comparison and matching. In this shared space, specific distance metrics (such as Euclidean distance or cosine similarity) are used to calculate the matching degree between different modalities. Two modalities are considered a match only when the distance between their features in the shared space is sufficiently small.

[0138] Then, for the matched data of different modalities, a self-attention mechanism is used to perform weighted fusion of multiple modalities such as images, text, and point clouds. This allows the model to dynamically adjust the contribution of the data according to the importance of different modalities. The matching space constructed in this way can improve the model's adaptability to complex datasets, not only improving the model's accuracy but also providing strong support for the effective integration of multimodal learning frameworks.

[0139] Through step S502, this embodiment successfully obtained a multimodal model by constructing a matching space, laying the foundation for subsequent prediction of customer terminal multimodal data and continuous optimization through the model.

[0140] In one embodiment of the annotation method based on multimodal models in this application, see [link to documentation]. Figure 6 It can also specifically include the following:

[0141] Step S601: Perform a difference comparison operation based on the predicted result and the set true result to determine the corresponding deviation value;

[0142] Step S602: Update the multimodal model based on the deviation value and the backpropagation algorithm to determine the updated multimodal model, wherein the set true result is determined by the annotator.

[0143] Optionally, in this embodiment, the purpose of this step is to iteratively optimize the model through a feedback mechanism to ensure that the model can learn quickly based on real-time data and annotator feedback.

[0144] Optionally, in this embodiment, the multimodal model is deployed on the annotation terminal used by the annotator. The intelligent agent platform (AIAgent) of each terminal observes the annotator's annotation behavior in real time and continuously compares it with the inference results of the multimodal model to continuously calibrate the model prediction results and achieve the effect of model training.

[0145] Optionally, in this embodiment, the model's prediction results are compared one by one with the annotators' annotation results, and the difference is calculated.

[0146] Optionally, for example, taking 2D image data and 3D point cloud data as examples, the annotator manually annotates in 2D / 3D space respectively, extracting the ground truth values ​​of the targets in the corresponding spaces (for dynamic targets, the data results are generally 2D / 3D bounding boxes; for 2D bounding boxes, the format of the bounding box is the center point coordinates and the corresponding width and height, x1, y1, w, l; for static map elements, it is generally a polyline). The output format of the model's inference results matches the output format of the annotator's manual annotation results. The annotator's output results (x_l, y_l, w_l, h_l) and the model's inference results (x_m, y_m, w_m, h_m) are directly interpolated to obtain the difference between the two (x_l-x_m, y_l-y_m, w_l-w_m, h_l-h_m). The annotation results of the annotators are used as the true values. The deviations (x_l-x_m, y_l-y_m, w_l-w_m, h_l-h_m) between the model's prediction results and the annotation results of the annotators are compared in real time. Based on the backpropagation algorithm for training the neural network model, these deviations are passed to the model as feedback for model training and optimization.

[0147] Optionally, in this optimization step, after the model makes a prediction, the annotator verifies the prediction results of the model. If the verification fails, the annotation is manually adjusted. Each manual adjustment by the annotator will trigger the model training of the intelligent agent platform. The feedback iteration can accelerate the performance improvement of the model.

[0148] Understandably, for multimodal data (such as text data, image data, and point cloud 3D data), the AI ​​Agent (intelligent agent platform) also compares the manual annotations of the annotator with the inference results of the model. For the difference feedback of multimodal data, all difference calculations will be used as training samples to perform cross-modal model iteration, thereby optimizing the prediction ability of the multimodal large model.

[0149] Through step S602, this embodiment successfully performs iterative tuning of the modal model through a feedback mechanism, ensuring rapid convergence and optimization of model performance.

[0150] In one embodiment of the annotation method based on multimodal models in this application, see [link to documentation]. Figure 7 It can also specifically include the following:

[0151] Step S701: The annotator manually annotates the dynamic data in the client terminal data on the annotator terminal to determine the corresponding dynamic data enveloping frame;

[0152] Step S702: The annotator manually annotates the static data in the client terminal data on the annotator's terminal to determine the corresponding static data polyline;

[0153] Step S703: Determine the corresponding true result based on the dynamic data enveloping frame and the static data polyline.

[0154] Optionally, the actual result is obtained by the annotator annotating the client terminal data on the annotation terminal.

[0155] Specifically, annotators manually annotate data using terminal devices, including text, images (2D data), and 3D point cloud data. Every annotation action of the annotator is recorded in real time by the AIAgent.

[0156] Specifically, for various types of client terminal data (text, images, and 3D point clouds), there are two forms: dynamic and static. For dynamic targets, the data result is generally a 2D / 3D bounding box (for 2D bounding boxes, the format of the bounding box is the center point coordinates and the corresponding width and height, x1, y1, w, l); for static map elements, it is generally a polyline.

[0157] Through step S703, this embodiment successfully obtained the actual results annotated by the annotators, laying the foundation for subsequent comparison with the model's prediction results, and using a feedback mechanism to help the model quickly adapt to new data distributions and requirements, thereby ensuring rapid convergence and optimization of model performance.

[0158] To improve the efficiency and accuracy of data annotation, this application provides an embodiment of a multimodal model-based annotation apparatus for implementing all or part of the aforementioned multimodal model-based annotation method. See [link to embodiment]. Figure 8 The annotation device based on the multimodal model specifically includes the following components:

[0159] Data receiving module 10 is used to receive client terminal dataset, which includes text data, image data and three-dimensional point cloud data;

[0160] The multimodal model training module 20 is used to input the client terminal dataset into the data preprocessing module of the preset initial model, perform segmentation operations on the text data according to the set adaptive word segmentation algorithm to determine the corresponding text vector, perform segmentation operations on the image data according to the preset image segmentation algorithm to determine the corresponding target image region, foreground image region and background image region, perform high-definition feature extraction operations on the foreground image region to determine the corresponding foreground features, perform fuzzy feature extraction operations on the background image region to determine the corresponding background features, perform feature extraction operations on the target image region to determine the corresponding target features, perform vectorization operations on the target features, the foreground features and the background features to determine the corresponding image vector, perform multi-scale point cloud feature extraction operations on the three-dimensional point cloud data according to the preset multi-scale feature extraction network to determine the corresponding point cloud vector, and input the text vector, the image vector and the point cloud vector into the backbone network of the preset initial model to perform matching space construction operations to determine the corresponding multimodal model;

[0161] The prediction annotation module 30 is used to perform prediction annotation operations on the client terminal data according to the multimodal model, and determine the prediction results of the client terminal data output by the multimodal model, wherein the prediction results include predicted text annotations, predicted image annotations, and predicted 3D point cloud annotations.

[0162] As described above, the annotation device based on a multimodal model provided in this application can input a client terminal dataset into a preset initial model, segment text data according to a set adaptive word segmentation algorithm to obtain text vectors, segment image data according to a preset image segmentation algorithm, extract features from the segmented target image region, foreground image region, and background image region to obtain image vectors, extract multi-scale point cloud features from 3D point cloud data according to a preset multi-scale feature extraction network to obtain point cloud vectors, input the text vectors, image vectors, and point cloud vectors into the backbone network of the preset initial model to perform a matching space construction operation to determine the corresponding multimodal model, perform prediction annotation operations on the client terminal data according to the multimodal model, and determine the prediction result of the client terminal data output by the multimodal model, thereby improving the efficiency and accuracy of data annotation.

[0163] From a hardware perspective, in order to improve the efficiency and accuracy of data annotation, this application provides an embodiment of an electronic device for implementing all or part of the annotation method based on the multimodal model, wherein the electronic device specifically includes the following:

[0164] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the multimodal model-based annotation method and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the multimodal model-based annotation method in the present embodiment, and the content of the embodiments of the multimodal model-based annotation method is incorporated herein, and repeated parts will not be described again.

[0165] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0166] In practical applications, the annotation method based on the multimodal model can be partially executed on the electronic device side as described above, or all operations can be completed in the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0167] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0168] Figure 9 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 9 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 9 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0169] In one embodiment, the annotation method based on the multimodal model can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0170] Step S101: Receive client terminal dataset, which includes text data, image data, and 3D point cloud data;

[0171] Step S102: Input the client terminal dataset into the data preprocessing module of the preset initial model, perform segmentation operation on the text data according to the set adaptive word segmentation algorithm to determine the corresponding text vector, perform segmentation operation on the image data according to the preset image segmentation algorithm to determine the corresponding target image region, foreground image region and background image region, perform high-definition feature extraction operation on the foreground image region to determine the corresponding foreground features, perform fuzzy feature extraction operation on the background image region to determine the corresponding background features, perform feature extraction operation on the target image region to determine the corresponding target features, perform vectorization operation on the target features, the foreground features and the background features to determine the corresponding image vector, perform multi-scale point cloud feature extraction operation on the three-dimensional point cloud data according to the preset multi-scale feature extraction network to determine the corresponding point cloud vector, input the text vector, the image vector and the point cloud vector into the backbone network of the preset initial model to perform matching space construction operation to determine the corresponding multimodal model;

[0172] Step S103: Perform prediction and annotation operations on the client terminal data according to the multimodal model, and determine the prediction results of the client terminal data output by the multimodal model, wherein the prediction results include predicted text annotations, predicted image annotations, and predicted 3D point cloud annotations.

[0173] As described above, the electronic device provided in this application embodiment inputs a client terminal dataset into a preset initial model, segments text data according to a set adaptive word segmentation algorithm to obtain text vectors, segments image data according to a preset image segmentation algorithm, extracts features from the segmented target image region, foreground image region, and background image region to obtain image vectors, extracts multi-scale point cloud features from 3D point cloud data according to a preset multi-scale feature extraction network to obtain point cloud vectors, inputs the text vectors, image vectors, and point cloud vectors into the backbone network of the preset initial model to perform a matching space construction operation, determines the corresponding multimodal model, performs prediction annotation operations on the client terminal data according to the multimodal model, and determines the prediction result of the client terminal data output by the multimodal model, thereby improving the efficiency and accuracy of data annotation.

[0174] In another implementation, the annotation method based on the multimodal model can be configured separately from the central processing unit 9100. For example, the annotation method based on the multimodal model can be configured as a chip connected to the central processing unit 9100, and the function of the annotation method based on the multimodal model can be realized through the control of the central processing unit.

[0175] like Figure 9 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 9 All components shown; in addition, the electronic device 9600 may also include Figure 9 For components not shown, please refer to existing technologies.

[0176] like Figure 9 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0177] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0178] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0179] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0180] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0181] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0182] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is also coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored sound via the speaker 9131.

[0183] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the multimodal model-based annotation method with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the multimodal model-based annotation method with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0184] Step S101: Receive client terminal dataset, which includes text data, image data, and 3D point cloud data;

[0185] Step S102: Input the client terminal dataset into the data preprocessing module of the preset initial model, perform segmentation operation on the text data according to the set adaptive word segmentation algorithm to determine the corresponding text vector, perform segmentation operation on the image data according to the preset image segmentation algorithm to determine the corresponding target image region, foreground image region and background image region, perform high-definition feature extraction operation on the foreground image region to determine the corresponding foreground features, perform fuzzy feature extraction operation on the background image region to determine the corresponding background features, perform feature extraction operation on the target image region to determine the corresponding target features, perform vectorization operation on the target features, the foreground features and the background features to determine the corresponding image vector, perform multi-scale point cloud feature extraction operation on the three-dimensional point cloud data according to the preset multi-scale feature extraction network to determine the corresponding point cloud vector, input the text vector, the image vector and the point cloud vector into the backbone network of the preset initial model to perform matching space construction operation to determine the corresponding multimodal model;

[0186] Step S103: Perform prediction and annotation operations on the client terminal data according to the multimodal model, and determine the prediction results of the client terminal data output by the multimodal model, wherein the prediction results include predicted text annotations, predicted image annotations, and predicted 3D point cloud annotations.

[0187] As described above, the computer-readable storage medium provided in this application embodiment improves the efficiency and accuracy of data annotation by inputting a client terminal dataset into a preset initial model, segmenting text data according to a set adaptive word segmentation algorithm to obtain text vectors, segmenting image data according to a preset image segmentation algorithm, extracting features from the segmented target image region, foreground image region, and background image region to obtain image vectors, extracting multi-scale point cloud features from 3D point cloud data according to a preset multi-scale feature extraction network to obtain point cloud vectors, inputting the text vectors, image vectors, and point cloud vectors into the backbone network of the preset initial model for matching space construction to determine the corresponding multimodal model, performing prediction annotation operations on the client terminal data according to the multimodal model, and determining the prediction results of the client terminal data output by the multimodal model.

[0188] Embodiments of this application also provide a computer program product capable of implementing all steps of the multimodal model-based annotation method with the execution subject being a server or client in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the multimodal model-based annotation method. For example, the computer program / instruction implements the following steps:

[0189] Step S101: Receive client terminal dataset, which includes text data, image data, and 3D point cloud data;

[0190] Step S102: Input the client terminal dataset into the data preprocessing module of the preset initial model, perform segmentation operation on the text data according to the set adaptive word segmentation algorithm to determine the corresponding text vector, perform segmentation operation on the image data according to the preset image segmentation algorithm to determine the corresponding target image region, foreground image region and background image region, perform high-definition feature extraction operation on the foreground image region to determine the corresponding foreground features, perform fuzzy feature extraction operation on the background image region to determine the corresponding background features, perform feature extraction operation on the target image region to determine the corresponding target features, perform vectorization operation on the target features, the foreground features and the background features to determine the corresponding image vector, perform multi-scale point cloud feature extraction operation on the three-dimensional point cloud data according to the preset multi-scale feature extraction network to determine the corresponding point cloud vector, input the text vector, the image vector and the point cloud vector into the backbone network of the preset initial model to perform matching space construction operation to determine the corresponding multimodal model;

[0191] Step S103: Perform prediction and annotation operations on the client terminal data according to the multimodal model, and determine the prediction results of the client terminal data output by the multimodal model, wherein the prediction results include predicted text annotations, predicted image annotations, and predicted 3D point cloud annotations.

[0192] As described above, the computer program product provided in this application embodiment inputs a client terminal dataset into a preset initial model, segments text data according to a set adaptive word segmentation algorithm to obtain text vectors, segments image data according to a preset image segmentation algorithm, extracts features from the segmented target image region, foreground image region, and background image region to obtain image vectors, extracts multi-scale point cloud features from 3D point cloud data according to a preset multi-scale feature extraction network to obtain point cloud vectors, inputs the text vectors, image vectors, and point cloud vectors into the backbone network of the preset initial model to perform a matching space construction operation, determines the corresponding multimodal model, performs prediction annotation operations on the client terminal data according to the multimodal model, and determines the prediction result of the client terminal data output by the multimodal model, thereby improving the efficiency and accuracy of data annotation.

[0193] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0194] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0195] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0196] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0197] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multi-modal model-based labeling method, characterized in that, The method is applied to an agent labeling platform in communication connection with a preset labeler terminal, and the method comprises the following steps: Receiving a client terminal data set, the client terminal data set comprising text data, image data and three-dimensional point cloud data; Inputting the client terminal data set into a data preprocessing module of a preset initial model, performing segmentation operation on the text data according to a set adaptive word segmentation algorithm, determining a corresponding text vector, performing segmentation operation on the image data according to a preset image segmentation algorithm, determining a corresponding target image region, a foreground image region and a background image region, performing high-definition feature extraction operation on the foreground image region, determining a corresponding foreground feature, performing fuzzy feature extraction operation on the background image region, determining a corresponding background feature, performing feature extraction operation on the target image region, determining a corresponding target feature, performing vectorization operation on the target feature, the foreground feature and the background feature, determining a corresponding image vector, performing multi-scale point cloud feature extraction operation on the three-dimensional point cloud data according to a preset multi-scale feature extraction network, determining a corresponding point cloud vector, inputting the text vector, the image vector and the point cloud vector into a backbone network of the preset initial model, performing distance measurement operation on the text vector, the image vector and the point cloud vector according to a preset shared space, determining a corresponding alignment degree, performing matching space construction operation on the text vector, the image vector and the point cloud vector according to the alignment degree, determining a corresponding multi-modal vector mapping relationship, and determining a corresponding multi-modal model according to the multi-modal vector mapping relationship; Performing prediction labeling operation on the client terminal data according to the multi-modal model, determining a prediction result of the client terminal data output by the multi-modal model, performing difference comparison operation on the prediction result and a set true result, determining a corresponding deviation value, updating the multi-modal model according to the deviation value and a back propagation algorithm, and determining an updated multi-modal model, wherein the prediction result comprises predicted text labeling, predicted image labeling and predicted three-dimensional point cloud labeling, and the set true result is determined by the labeler terminal. 2.The method of claim 1, wherein, Before the step of performing segmentation on the text data according to the set word segmentation algorithm and determining a corresponding text vector, the following steps are further included: Collecting a historical text data set, performing word segmentation operation on the historical text data set, and determining a corresponding minimum text subunit; Inputting the minimum text subunit into a preset long short-term network for model training, and determining a dependency relationship between the minimum text subunits output by the long short-term network; According to the dependency relationship, a preset validation set dependency relationship and a preset cross-entropy loss function, a corresponding loss function is determined, and the parameters of the long short-term network are updated according to the optimal loss function and a back update algorithm, and a corresponding adaptive word segmentation algorithm is determined. 3.The method of claim 1, wherein, The step of performing segmentation on the text data according to the set word segmentation algorithm and determining a corresponding text vector comprises the following steps: According to the set word segmentation algorithm, the text data is analyzed to determine a corresponding text data segmentation point, and the text data is segmented according to the text data segmentation point to determine a corresponding optimal text subunit; According to a preset word vector model, the optimal text subunit is vectorized to determine a corresponding text vector. 4.The method of claim 1, wherein, The multi-scale point cloud feature extraction network is used to perform multi-scale point cloud feature extraction on the three-dimensional point cloud data to determine a corresponding point cloud vector, including: According to a preset far point sampling algorithm, the three-dimensional point cloud data is sampled to determine a corresponding centroid point, and a preset multi-scale radius is used to construct a neighborhood for the centroid point to determine a corresponding multi-scale neighborhood; The multi-scale neighborhood is subjected to feature extraction, and the multi-scale neighborhood features obtained after the feature extraction are subjected to feature fusion to determine a multi-scale feature corresponding to the centroid point, and the multi-scale feature is subjected to vectorization to determine a corresponding point cloud vector. 5.The method of claim 1, wherein, Before the difference comparison operation according to the prediction result and the set true result is performed to determine a corresponding deviation value, including: The annotator manually annotates the dynamic data in the customer terminal data on the annotator terminal to determine a corresponding dynamic data envelope box; The annotator manually annotates the static data in the customer terminal data on the annotator terminal to determine a corresponding static data polyline; The true result is determined according to the dynamic data envelope box and the static data polyline.

6. A labeling apparatus based on a multi-modal model, characterized by, The device implements the steps of the annotation method based on the multi-modal model according to any one of claims 1 to 5, including: A data receiving module is configured to receive a customer terminal data set, the customer terminal data set including text data, image data, and three-dimensional point cloud data; A multi-modal model training module is configured to input the customer terminal data set into a data preprocessing module of a preset initial model, perform segmentation on the text data according to a set adaptive word segmentation algorithm to determine a corresponding text vector, perform segmentation on the image data according to a preset image segmentation algorithm to determine a corresponding target image region, foreground image region, and background image region, perform high-definition feature extraction on the foreground image region to determine a corresponding foreground feature, perform fuzzy feature extraction on the background image region to determine a corresponding background feature, perform feature extraction on the target image region to determine a corresponding target feature, perform vectorization on the target feature, the foreground feature, and the background feature to determine a corresponding image vector, and perform multi-scale point cloud feature extraction on the three-dimensional point cloud data according to a preset multi-scale feature extraction network to determine a corresponding point cloud vector; A prediction annotation module is configured to perform prediction annotation on the customer terminal data according to the multi-modal model to determine a prediction result of the customer terminal data output by the multi-modal model, wherein the prediction result includes a predicted text annotation, a predicted image annotation, and a predicted three-dimensional point cloud annotation.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the multi-modal model based labeling method in any one of claims 1-5 when executing the program.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the multi-modal model based labeling method in any one of claims 1-5.

Images