Model training method and device, model, storage medium and terminal equipment
By aligning the features of 3D point cloud data with those of text and image data, pseudo-label training samples are generated, which solves the problem that 3D environmental information is not fully captured in existing technologies and achieves better multimodal data adaptability and versatility.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING CO WHEELS TECH CO LTD
- Filing Date
- 2024-10-28
- Publication Date
- 2026-04-28
AI Technical Summary
Existing technologies cannot effectively process 3D point cloud data, resulting in the failure to fully capture complex information in the 3D environment, and the poor adaptability and versatility of the model when dealing with multimodal data.
By acquiring training data of the first, second, and third modalities with the same semantic information, matching prediction results using the first and second pre-trained models, generating pseudo-label training samples, and generating pseudo-label training samples of the third modality through modality transformation, the first and second networks are trained, and feature alignment is achieved by combining data features of different modalities.
It improves the model's ability to understand multimodal scenes, enhances the model's adaptability and versatility, and improves 3D segmentation results.
Smart Images
Figure CN121936620A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of deep learning technology, specifically to a model training method, a model training device, a model, a machine-readable storage medium, and a terminal device. Background Technology
[0002] The Visual Language Model (VLM) is a large-scale model for images and text. By aligning the feature spaces of images and text, it is widely used in various tasks such as autonomous driving, image processing and generation, and multimodal understanding. Specific applications include Visual Question Answering (VQA) and Image Captioning. Currently, traditional visual language models primarily focus on feature space alignment for 2D images and text, but have limited processing capabilities for 3D point cloud data, failing to fully capture the complex information in 3D environments. Due to the scarcity of 3D point cloud and text alignment data, current large-scale 3D point cloud-text design models cannot effectively understand 3D scenes in autonomous driving scenarios. Existing technologies lack a unified framework to simultaneously process 2D images, 3D point clouds, and text information, resulting in poor adaptability and versatility of models when dealing with multimodal data. Summary of the Invention
[0003] The purpose of this application is to provide a model training method, apparatus, model, storage medium, and terminal device to solve the problem that existing large models cannot fully capture complex information in the 3D environment, resulting in poor adaptability and versatility of the model when dealing with multimodal data.
[0004] To achieve the above objectives, a first aspect of this application provides a model training method, wherein the large model includes at least a first network and a second network, the first network and the second network being used to process data of different modalities, the method comprising:
[0005] Obtain the original training samples, which include first modality training data, second modality training data and third modality training data with the same semantic information, wherein the first modality training data, the second modality training data and the third modality training data are data of different modalities;
[0006] The first pre-trained model outputs a first prediction result based on the first modality training data and the second modality training data; the second pre-trained model outputs a second prediction result based on the second modality training data.
[0007] The first prediction result and the second prediction result are matched, and a first pseudo-label training sample corresponding to the second modality training data is generated based on the obtained matching result. The first pseudo-label training sample is modally transformed to obtain a second pseudo-label training sample corresponding to the third modality training data.
[0008] The first network is trained using the second modality training data and the corresponding first pseudo-label training samples, and the second network is trained using the third modality training data and the corresponding second pseudo-label training samples.
[0009] In this embodiment of the application, for each original training sample, the method further includes:
[0010] Using the second modality training data as input, the second pre-trained model extracts the first data feature of the second modality training data; using the second modality training data as input, the first network extracts the second data feature of the second modality training data; and using the third modality training data as input, the second network extracts the third data feature of the third modality training data.
[0011] Calculate the first similarity between the first data feature and the second data feature, and calculate the second similarity between the first data feature and the second data feature;
[0012] A first loss function of the first network is determined based on the first similarity, and a second loss function of the second network is determined based on the second similarity.
[0013] Training the first network based on the second modality training data and the corresponding first pseudo-label training samples, and training the second network based on the third modality training data and the corresponding second pseudo-label training samples, includes:
[0014] Using the first loss function as the optimization objective of the first network, the first network is trained using the second modality training data and the corresponding first pseudo-label training samples; and
[0015] The second loss function is used as the optimization objective of the second network, and the second network is trained using the second modality training data and the corresponding second pseudo-label training samples.
[0016] In this embodiment of the application, training the first network based on the second modality training data and the corresponding first pseudo-label training samples, and training the second network based on the third modality training data and the corresponding second pseudo-label training samples, further includes:
[0017] Using total cross-entropy loss as the optimization objective for the first network and the second network, the first network is trained using the second modality training data and the corresponding first pseudo-label training samples, and the second network is trained using the third modality training data and the corresponding second pseudo-label training samples.
[0018] The total cross-entropy is obtained through the following steps:
[0019] Using the second modality training data or the corresponding first pseudo-label training sample as input, the prediction result of the first network is obtained through the first network, and the cross-entropy loss of the first network is calculated based on the prediction result of the first network.
[0020] Using the third modality training data or the corresponding second pseudo-label training samples as input, the prediction result of the second network is obtained through the second network, and the cross-entropy loss of the second network is calculated based on the prediction result of the second network.
[0021] The cross-entropy loss of the first network and the cross-entropy loss of the second network are weighted and averaged to obtain the total cross-entropy loss of the first network and the second network.
[0022] In this embodiment of the application, training the first network based on the second modality training data and the corresponding first pseudo-label training samples, and training the second network based on the third modality training data and the corresponding second pseudo-label training samples, further includes:
[0023] The first network is trained using the second modality training data and the corresponding first pseudo-label training samples until the first termination condition is met; and
[0024] The second network is trained using the second modality training data and the corresponding second pseudo-label training samples until the second termination condition is met.
[0025] The first termination condition includes reaching a first number of iterations or the prediction accuracy of the first network reaching a first threshold, and the second termination condition includes reaching a second number of iterations or the prediction accuracy of the second network reaching a second threshold.
[0026] In this embodiment of the application, the first prediction result includes first label information indicating whether each minimum processing unit of the second modality training data matches the semantic information, and the second prediction result includes second label information indicating whether each minimum processing unit of the second modality training data matches the semantic information; generating a first pseudo-label training sample corresponding to the second modality training data based on the obtained matching result includes:
[0027] For each smallest processing unit of the training data of the second modality, the first label information and the second label information are matched. If the first label information and the second label information are the same, the first label information is retained. If the first label information and the second label information are different, the first label information is replaced with the second label information.
[0028] The process continues until the label matching for each smallest processing unit of the second modality training data is completed, resulting in the first pseudo-label training sample corresponding to the second modality training data.
[0029] In this embodiment of the application, the first pre-trained model includes at least a first modality encoder and a second modality encoder. The first prediction result of the semantic information output by the first pre-trained model includes:
[0030] The first modality training data is subjected to feature extraction by the first modality encoder to obtain the fourth data feature of the first modality training data;
[0031] The second modality training data is subjected to feature extraction by the second modality encoder to obtain the fifth data feature of the second modality training data;
[0032] Perform an inner product operation on the fourth data feature and the fifth data feature to obtain a first prediction result that includes first label information representing whether each minimum processing unit of the second modality training data matches the semantic information.
[0033] In this embodiment of the application, the first modality training data is text data, the second modality training data is image data, and the third modality training data is point cloud data;
[0034] The first pre-trained model is a cross-modal model based on contrasting text-image pairs, and the second pre-trained model is an image segmentation model.
[0035] A second aspect of this application provides a model training apparatus that applies the above-described model training method, the apparatus comprising:
[0036] The training data acquisition module is configured to acquire raw training samples, which include first modality training data, second modality training data and third modality training data with the same semantic information, wherein the first modality training data, the second modality training data and the third modality training data are data of different modalities;
[0037] The training data processing module is configured to: output a first prediction result from a first pre-trained model based on the first modality training data and the second modality training data; output a second prediction result from a second pre-trained model based on the second modality training data; match the first prediction result and the second prediction result; generate a first pseudo-label training sample corresponding to the second modality training data based on the obtained matching result; and perform modality transformation on the first pseudo-label training sample to obtain a second pseudo-label training sample corresponding to the third modality training data.
[0038] The model training module is configured to train the first network based on the second modality training data and the corresponding first pseudo-label training samples, and to train the second network based on the third modality training data and the corresponding second pseudo-label training samples.
[0039] In a third aspect, this application provides a model, which is trained using the model training method described above.
[0040] In a fourth aspect, this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the model training method described above.
[0041] In a fifth aspect, this application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the model training method described above.
[0042] Through the above technical solution, this application uses a first pre-trained model based on modality comparison to predict the first and second modal training data of different modalities. The predicted results are then matched with the predicted results of the second pre-trained model for the second modal training data to generate pseudo-labeled training samples for that modality. Then, pseudo-labeled training samples for the third modality are generated through modality transformation. Based on the generated pseudo-labeled training samples of different modalities, the corresponding deep learning network is trained. This effectively combines the data features of different modalities to train the deep learning network, realizes the alignment of features of different modalities, effectively improves the adaptability and versatility of the model, and enhances the model's ability to understand multimodal scenarios.
[0043] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description
[0044] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:
[0045] Figure 1 A flowchart illustrating a model training method according to an embodiment of this application is shown schematically.
[0046] Figure 2 A schematic diagram of a large model structure according to an embodiment of this application is shown;
[0047] Figure 3 A schematic block diagram of a model training apparatus according to an embodiment of this application is shown.
[0048] Figure 4 A schematic diagram of the structure of a terminal device according to an embodiment of this application is shown.
[0049] Explanation of reference numerals in the attached figures
[0050] 10 - Terminal device, 100 - Processor, 101 - Memory, 102 - Computer program. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0052] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.
[0053] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0054] Figure 1 This illustration schematically depicts a model training method according to an embodiment of this application, particularly a method for training a large multimodal scene understanding model. The large model includes at least a first network and a second network, which are used to process data from different modalities, respectively. The method includes:
[0055] S100. Obtain the original training samples. The original training samples include first modality training data, second modality training data and third modality training data with the same semantic information. The first modality training data, the second modality training data and the third modality training data are data of different modalities.
[0056] S200: Call the first pre-trained model based on modality comparison, take the first modality training data and the second modality training data as input, and output the first prediction result of semantic information through the first pre-trained model. Call the second pre-trained model, take the second modality training data as input, and output the second prediction result of semantic information through the second pre-trained model. Match the first prediction result and the second prediction result, generate the first pseudo-label training sample corresponding to the second modality training data based on the obtained matching result, and obtain the second pseudo-label training sample corresponding to the third modality training data after modality transformation of the first pseudo-label training sample.
[0057] S300. The first network is trained based on the second modality training data and the corresponding first pseudo-label training samples, and the second network is trained based on the third modality training data and the corresponding second pseudo-label training samples, so as to obtain the trained model when the training termination condition is met.
[0058] Thus, this application uses a first pre-trained model based on modality comparison to predict the first and second modal training data of different modalities. The predicted results are then matched with the predicted results of the second pre-trained model on the second modal training data to generate pseudo-labeled training samples for that modality. Then, a third modal pseudo-labeled training sample is generated through modality transformation. Based on the generated pseudo-labeled training samples of different modalities, the corresponding deep learning network is trained. This effectively combines the data features of different modalities to train the deep learning network, achieving the alignment of features of different modalities. This effectively improves the adaptability and versatility of model training and enhances the model's understanding of multimodal scenarios.
[0059] As we can understand, a modality refers to a way of expressing or perceiving things. Every source or form of information can be called a modality. For example, humans have touch, hearing, vision, and smell; information media include speech, video, and text; different sensors, such as radar, infrared, and accelerometers, can all be called modalities. Multimodality refers to expressing or perceiving things from multiple modalities. Multimodality can be categorized into homogeneous modalities, such as pictures taken from two different cameras, or heterogeneous modalities, such as pictures and text.
[0060] In this embodiment, the first network is a 2D image segmentation network, and the second network is a point cloud data segmentation network. In step S100, the first modality training data is text data, the second modality training data is image data, and the third modality training data is point cloud data. The first, second, and third modality training data have the same semantic information. For example, for an original training sample whose semantic information indicates the target to be detected is a cone, its first modality training data is text information describing the cone, the second modality training data is image data of the cone collected by a camera, and the third modality training data is the collected point cloud data of the cone. For example, point cloud data can be collected using LiDAR or millimeter-wave radar, etc., which is not limited here. In this application, each original training sample may include multiple second modality training data, or one second modality training data may include multiple images. For example, in an autonomous driving scenario, it may include image data of the vehicle under multiple visions, including the vehicle's 6V image data, i.e., image data of the vehicle in the forward vision, the rear vision, and four side visions. The third modality training data consists of point cloud data of the target to be detected.
[0061] In step S200, the first pre-trained model in this application embodiment can adopt the existing CLIP (Contrastive Language-Image Pre-Training) model, which is a pre-trained neural network model for matching images and text. It aligns the feature spaces of images and text through contrastive learning, thereby effectively handling various multimodal tasks. The second pre-trained model adopts the existing SAM (Segment Anything Model) model, which is a large-scale image segmentation model.
[0062] like Figure 2 As shown, during large-scale model training, the pre-trained CLIP model is first invoked. The CLIP model is input using first-modality training data, second-modality training data, and visual prompt information from the original training samples with the same semantic information. The visual prompt can be understood as an input cue for the visual processing task; it is typically used to guide and specify the object or region the model should focus on. In other words, the visual prompt is an image-based input signal or instruction that helps the model understand the task requirements and generate the corresponding output. The first-modality training data, i.e., the text information, can be descriptive information about the target to be detected, such as vehicles, zebra crossings, or drivable areas. The second-modality training data consists of image data including the target in at least one visual perspective.
[0063] In this embodiment, the first pre-trained model includes at least a first modal encoder and a second modal encoder. The first pre-trained model outputs a first prediction result for semantic information, including: extracting features from the first modal training data using the first modal encoder to obtain a fourth data feature of the first modal training data; extracting features from the second modal training data using the second modal encoder to obtain a fifth data feature of the second modal training data; and performing an inner product operation on the fourth and fifth data features to obtain a first prediction result including first label information indicating whether each smallest processing unit of the second modal training data matches the semantic information. Specifically, taking the CLIP model as an example, the CLIP model's text encoder extracts text features from the input text information, and the CLIP model's image encoder extracts image features from the input image data. By performing an inner product operation on the text features and image features, a prediction result, i.e., a clip mask, is obtained for each smallest processing unit of the input image data. In this application, the smallest processing unit for image data can be each pixel of the image or a region consisting of N pixels. For example, the text information is a bucket shape, and the image data is an image including the bucket shape. The CLIP model is used to extract features from both the text information and the image data. The similarity between the text features and the image features of each smallest processing unit is obtained through inner product operations. This yields a prediction result as to whether each smallest processing unit belongs to the bucket shape. For instance, if a smallest processing unit is predicted to belong to the bucket shape, its label information is set to 1; otherwise, its label information is set to 0. The label information of each pixel constitutes the first prediction result. The inner product operation on the text features and image features is existing technology and will not be elaborated upon here.
[0064] Using the second modality training data, such as images including bucket shapes, as input to the SAM model, the SAM model performs image segmentation on the input image data, obtaining a second prediction result that includes second label information indicating whether each smallest processing unit of the input second modality training data matches semantic information. For example, the SAM model can obtain a segmentation mask for bucket shapes in the input image. It is understood that the segmentation mask is used to accurately separate objects in the image from the background. It achieves fine-grained division of image regions by classifying and labeling each pixel. Each pixel is assigned a label to indicate whether it belongs to the foreground or background, or to a different object category. For example, if the current pixel belongs to a bucket shape, its label information is set to 1; otherwise, it is set to 1. The two-dimensional matrix formed by the label information is the segmentation mask. In this application, the segmentation mask constitutes the second prediction result.
[0065] After obtaining the label information of the input image for a given semantic information at each pixel through the CLIP model and the label information of the same input image for the same given semantic information at each pixel through the SAM model, this application further denoises the output of the CLIP model based on the output of the SAM model, and uses the denoised prediction results to generate new pseudo-label data as training data to train the 2D image segmentation network and the point cloud data segmentation network. This achieves the alignment of text information with image data and point cloud data at the region level, such as at each pixel, thereby improving the training effect of the 2D image segmentation network and the point cloud data segmentation network.
[0066] Specifically, in this embodiment of the application, generating a first pseudo-label training sample corresponding to the second modality training data based on the obtained matching result includes:
[0067] S210. For each smallest processing unit of the second modality training data, perform label matching between the first label information and the second label information. If the first label information and the second label information are the same, retain the first label information; if the first label information and the second label information are different, replace the first label information with the second label information. For example, first, match the pixels in the output of the CLIP model with the pixels in the output of the SAM model. Since the input is the same image data, the pixel matching can be done directly. For pixel 1, if it is labeled as 1 in the output of the CLIP model, determine whether pixel 1 is labeled as 1 in the output of the SAM model. If it is labeled as 1 in the output of the SAM model, retain its label information, that is, keep the label information of pixel 1 as 1; if it is labeled as 0 in the output of the SAM model, set the label information of pixel 1 to 0.
[0068] S211. Continue until label matching is completed for each smallest processing unit of the second modality training data, obtaining the first pseudo-label training sample corresponding to the second modality training data. Repeat the above steps until matching is completed for each pixel of the input image, generating a labeled training sample corresponding to the input image with the updated label information, that is, generating a new labeled training sample with the input image and the new label information.
[0069] In this embodiment, after modal transformation of the first pseudo-label training sample, a second pseudo-label training sample corresponding to the third modality training data is obtained. This includes: projecting the third modality training data, i.e., point cloud data, into image space. For example, the projection of point cloud data to a 2D image can be achieved through multi-view input images, camera intrinsic parameters, and extrinsic parameters, thereby converting each point in the three-dimensional point cloud data into two-dimensional image coordinates, i.e., one point cloud corresponds to one pixel in the image. It is understood that projecting point cloud data to an image is existing technology, and it will not be elaborated here. After converting the point cloud data to two-dimensional image coordinates, the label information updated by updating the CLIP model output through the SAM model output is matched with the converted point cloud data to obtain labeled point cloud data training samples.
[0070] In step S300, in this embodiment of the application, for each original training sample, the method further includes:
[0071] S310. Using the second modality training data as input, the second pre-trained model extracts the first data features of the second modality training data; using the second modality training data as input, the first network extracts the second data features of the second modality training data; and using the third modality training data as input, the second network extracts the third data features of the third modality training data. The first and second data features are image features, and the third data feature is a point cloud feature; that is, the image features extracted by the SAM model from the input image data, the image features extracted by the 2D image segmentation network from the input image data, and the point cloud features extracted by the point cloud data segmentation network from the input point cloud data are obtained respectively.
[0072] S311. Calculate the first similarity between the first data feature and the second data feature, and calculate the second similarity between the first data feature and the second data feature. In this application, the calculation of the first similarity and the second similarity are both calculations of the cosine similarity between corresponding features. The calculation of cosine similarity is prior art and will not be described in detail here.
[0073] S312. Determine a first loss function for the first network based on the first similarity, and determine a second loss function for the second network based on the second similarity. That is, construct a loss function with the similarity between each feature as the optimization objective. For example, the first loss function and the second loss function can be constructed by taking the mean square error of the negative values of the first similarity or the second similarity.
[0074] Then, the first network is trained based on the second modality training data and the corresponding first pseudo-label training samples, and the second network is trained based on the third modality training data and the corresponding second pseudo-label training samples. This includes: training the first network using the first loss function as the optimization objective, through the second modality training data and the corresponding first pseudo-label training samples; and training the second network using the second loss function as the optimization objective, through the second modality training data and the corresponding second pseudo-label training samples. Using the original training data and the generated pseudo-label data as training samples, the 2D image segmentation network and the point cloud data segmentation network are trained. Network parameters are updated through backpropagation to ensure that the first similarity and the second similarity meet a preset threshold as much as possible. For example, if the similarity is set to 0-1, a higher similarity indicates greater similarity; therefore, during training, the optimization objective can be to keep the similarity as close to 1 as possible. This application performs feature distillation by calculating the similarity of features from different modalities. Based on the similarity of features from different modalities, it optimizes the 2D image segmentation network and the point cloud data segmentation network, which can effectively align text and image spaces. At the same time, it can effectively transfer the information of text and image space alignment to the 3D point cloud space, thereby improving the 3D segmentation effect.
[0075] While optimizing the 2D image segmentation network and point cloud data segmentation network based on the similarity between different modalities, this application also optimizes the 2D image segmentation network and point cloud data segmentation network through cross-training. Specifically, in the embodiments of this application, the first network is trained based on the second modality training data and the corresponding first pseudo-label training samples, and the second network is trained based on the third modality training data and the corresponding second pseudo-label training samples. The optimization also includes: using the total cross-entropy loss as the optimization objective for the first and second networks, training the first network with the second modality training data and the corresponding first pseudo-label training samples, and training the second network with the third modality training data and the corresponding second pseudo-label training samples.
[0076] The total cross-entropy is obtained through the following steps:
[0077] S320. Using the second modality training data or the corresponding first pseudo-label training samples as input, the prediction results of the first network are obtained through the first network, and the cross-entropy loss of the first network is calculated based on the prediction results of the first network. For the image branch, features are extracted from the original training samples or newly generated pseudo-label training samples through a 2D image segmentation network, and a class probability distribution is output. The cross-entropy loss between the output probability distribution and the actual label is calculated through the cross-entropy function, where the cross-entropy function is an existing function, which will not be elaborated here.
[0078] S321. Using the third modality training data or the corresponding second pseudo-label training samples as input, the second network is used to obtain the prediction results of the second network, and the cross-entropy loss of the second network is calculated based on the prediction results of the second network. For the point cloud branch, the point cloud data segmentation network is used to extract features from the original training samples or newly generated pseudo-label training samples, outputting a class probability distribution. The cross-entropy loss between the output probability distribution and the actual label is calculated using the cross-entropy function.
[0079] S322. The cross-entropy loss of the first network and the cross-entropy loss of the second network are weighted and averaged to obtain the total cross-entropy loss of the first and second networks. By weighting and averaging the cross-entropy losses obtained from the image branch and the point cloud branch, the total cross-entropy loss of the 2D image segmentation network and the point cloud data segmentation network can be obtained. The 2D image segmentation network and the point cloud data segmentation network are trained with the total cross-entropy loss as the optimization objective to minimize the total cross-entropy, thereby optimizing the 2D image segmentation network and the point cloud data segmentation network.
[0080] Simultaneously, this application also optimizes the network through self-training of a 2D image segmentation network and a point cloud data segmentation network. In this embodiment, training the first network based on the second modality training data and the corresponding first pseudo-label training samples, and training the second network based on the third modality training data and the corresponding second pseudo-label training samples, further includes:
[0081] S330. The first network is trained using the second modality training data and the corresponding first pseudo-label training samples until a first termination condition is met. The first termination condition includes reaching a first number of iterations or the prediction accuracy of the first network reaching a first threshold. This application, while performing feature distillation and cross-training, also self-trains the 2D image segmentation network using generated pseudo-label training samples until a preset maximum number of iterations is reached or the output accuracy of the 2D image segmentation network reaches a threshold, thus completing the training of the 2D image segmentation network.
[0082] S331. The second network is trained using the second modality training data and the corresponding second pseudo-label training samples until the second termination condition is met. Similarly, this application simultaneously trains the point cloud data segmentation network using the generated pseudo-label training samples until the preset maximum number of iterations is reached or the output accuracy of the point cloud data segmentation network reaches a threshold, thereby completing the training of the point cloud data segmentation network and obtaining the multimodal scene understanding large model training model.
[0083] like Figure 3 As shown, in a second aspect, this application provides a model training apparatus that applies the above-described model training method. The apparatus includes:
[0084] The training data acquisition module is configured to acquire raw training samples, which include first modality training data, second modality training data and third modality training data with the same semantic information. The first modality training data, the second modality training data and the third modality training data are data from different modalities.
[0085] The training data processing module is configured to: output a first prediction result from a first pre-trained model based on the first modality training data and the second modality training data; output a second prediction result from a second pre-trained model based on the second modality training data; match the first prediction result and the second prediction result; generate a first pseudo-label training sample corresponding to the second modality training data based on the obtained matching result; and perform modality transformation on the first pseudo-label training sample to obtain a second pseudo-label training sample corresponding to the third modality training data.
[0086] The model training module is configured to train the first network based on the second modality training data and the corresponding first pseudo-label training samples, and to train the second network based on the third modality training data and the corresponding second pseudo-label training samples, so as to obtain the trained model when the training termination condition is met.
[0087] In a third aspect, this application provides a model that is trained using the model training method described above.
[0088] In a fourth aspect, this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the model training method described above.
[0089] In a fifth aspect, this application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the model training method described above.
[0090] like Figure 4 The diagram shown is a schematic representation of a terminal device provided in an embodiment of this application. Figure 4 As shown, the terminal device 10 of this embodiment includes a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, it implements the steps in the above method embodiments. Alternatively, when the processor 100 executes the computer program 102, it implements the functions of each module / unit in the above device embodiments.
[0091] For example, computer program 102 may be divided into one or more modules / units, one or more of which are stored in memory 101 and executed by processor 100 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 102 in terminal device 10.
[0092] Terminal device 10 may be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 10 may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will understand that... Figure 4 This is merely an example of terminal device 10 and does not constitute a limitation on terminal device 10. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device may also include input / output devices, network access devices, buses, etc.
[0093] The processor 100 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0094] The memory 101 can be an internal storage unit of the terminal device 10, such as a hard disk or RAM of the terminal device 10. The memory 101 can also be an external storage device of the terminal device 10, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 10. Furthermore, the memory 101 can include both internal and external storage units of the terminal device 10. The memory 101 is used to store computer programs and other programs and data required by the terminal device 10. The memory 101 can also be used to temporarily store data that has been output or will be output.
[0095] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0096] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0099] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0100] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0101] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0102] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0103] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A model training method, characterized in that, The model includes at least a first network and a second network, wherein the first network and the second network are used to process data of different modalities, and the method includes: Obtain the original training samples, which include first modality training data, second modality training data and third modality training data with the same semantic information, wherein the first modality training data, the second modality training data and the third modality training data are data of different modalities; The first pre-trained model outputs a first prediction result based on the first modality training data and the second modality training data; the second pre-trained model outputs a second prediction result based on the second modality training data. The first prediction result and the second prediction result are matched, and a first pseudo-label training sample corresponding to the second modality training data is generated based on the obtained matching result. The first pseudo-label training sample is modally transformed to obtain a second pseudo-label training sample corresponding to the third modality training data. The first network is trained using the second modality training data and the corresponding first pseudo-label training samples, and the second network is trained using the third modality training data and the corresponding second pseudo-label training samples.
2. The model training method according to claim 1, characterized in that, For each original training sample, the method further includes: Using the second modality training data as input, the second pre-trained model extracts the first data feature of the second modality training data; using the second modality training data as input, the first network extracts the second data feature of the second modality training data; and using the third modality training data as input, the second network extracts the third data feature of the third modality training data. Calculate the first similarity between the first data feature and the second data feature, and calculate the second similarity between the first data feature and the second data feature; A first loss function of the first network is determined based on the first similarity, and a second loss function of the second network is determined based on the second similarity. Training the first network based on the second modality training data and the corresponding first pseudo-label training samples, and training the second network based on the third modality training data and the corresponding second pseudo-label training samples, includes: Using the first loss function as the optimization objective of the first network, the first network is trained using the second modality training data and the corresponding first pseudo-label training samples; and The second loss function is used as the optimization objective of the second network, and the second network is trained using the second modality training data and the corresponding second pseudo-label training samples.
3. The model training method according to claim 2, characterized in that, Training the first network based on the second modality training data and the corresponding first pseudo-label training samples, and training the second network based on the third modality training data and the corresponding second pseudo-label training samples, further includes: Using total cross-entropy loss as the optimization objective for the first network and the second network, the first network is trained using the second modality training data and the corresponding first pseudo-label training samples, and the second network is trained using the third modality training data and the corresponding second pseudo-label training samples. The total cross-entropy is obtained through the following steps: Using the second modality training data or the corresponding first pseudo-label training sample as input, the prediction result of the first network is obtained through the first network, and the cross-entropy loss of the first network is calculated based on the prediction result of the first network. Using the third modality training data or the corresponding second pseudo-label training samples as input, the prediction result of the second network is obtained through the second network, and the cross-entropy loss of the second network is calculated based on the prediction result of the second network. The cross-entropy loss of the first network and the cross-entropy loss of the second network are weighted and averaged to obtain the total cross-entropy loss of the first network and the second network.
4. The model training method according to claim 3, characterized in that, Training the first network based on the second modality training data and the corresponding first pseudo-label training samples, and training the second network based on the third modality training data and the corresponding second pseudo-label training samples, further includes: The first network is trained using the second modality training data and the corresponding first pseudo-label training samples until the first termination condition is met; and The second network is trained using the second modality training data and the corresponding second pseudo-label training samples until the second termination condition is met. The first termination condition includes reaching a first number of iterations or the prediction accuracy of the first network reaching a first threshold, and the second termination condition includes reaching a second number of iterations or the prediction accuracy of the second network reaching a second threshold.
5. The model training method according to claim 1, characterized in that, The first prediction result includes first label information indicating whether each minimum processing unit of the second modality training data matches the semantic information; the second prediction result includes second label information indicating whether each minimum processing unit of the second modality training data matches the semantic information. Based on the obtained matching results, a first pseudo-label training sample corresponding to the second modality training data is generated, including: For each smallest processing unit of the training data of the second modality, the first label information and the second label information are matched. If the first label information and the second label information are the same, the first label information is retained. If the first label information and the second label information are different, the first label information is replaced with the second label information. The process continues until the label matching for each smallest processing unit of the second modality training data is completed, resulting in the first pseudo-label training sample corresponding to the second modality training data.
6. The model training method according to claim 5, characterized in that, The first pre-trained model includes at least a first modality encoder and a second modality encoder. The first pre-trained model outputs a first prediction result for the semantic information, including: The first modality training data is subjected to feature extraction by the first modality encoder to obtain the fourth data feature of the first modality training data; The second modality training data is subjected to feature extraction by the second modality encoder to obtain the fifth data feature of the second modality training data; Perform an inner product operation on the fourth data feature and the fifth data feature to obtain a first prediction result that includes first label information representing whether each minimum processing unit of the second modality training data matches the semantic information.
7. The model training method according to any one of claims 1-6, characterized in that, The first modality training data is text data, the second modality training data is image data, and the third modality training data is point cloud data; The first pre-trained model is a cross-modal model based on contrasting text-image pairs, and the second pre-trained model is an image segmentation model.
8. A model training apparatus, employing the model training method according to any one of claims 1-7, characterized in that, The device includes: The training data acquisition module is configured to acquire raw training samples, which include first modality training data, second modality training data and third modality training data with the same semantic information, wherein the first modality training data, the second modality training data and the third modality training data are data of different modalities; The training data processing module is configured to: output a first prediction result from a first pre-trained model based on the first modality training data and the second modality training data; output a second prediction result from a second pre-trained model based on the second modality training data; match the first prediction result and the second prediction result; generate a first pseudo-label training sample corresponding to the second modality training data based on the obtained matching result; and perform modality transformation on the first pseudo-label training sample to obtain a second pseudo-label training sample corresponding to the third modality training data. The model training module is configured to train the first network based on the second modality training data and the corresponding first pseudo-label training samples, and to train the second network based on the third modality training data and the corresponding second pseudo-label training samples.
9. A model, characterized in that, The model is trained using the model training method described in any one of claims 1-7.
10. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the model training method as described in any one of claims 1-7.
11. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the model training method as described in any one of claims 1-7.