Image classification method and device, computer device, storage medium and program product

Abstract

Embodiments of the present disclosure relate to an image classification method and device, computer equipment, a storage medium and a program product. The method comprises: obtaining multi-modal data; determining a target neural network model from a plurality of pre-trained neural network models; inputting the multi-modal data into the target neural network model for classification processing to determine a target category corresponding to the multi-modal data; wherein each neural network model is obtained by training based on a plurality of labeled training data sets and a plurality of unlabeled training data sets, and each of the labeled training data sets and the unlabeled training data sets comprises multi-modal training data, and the unlabeled training data set is a training data set obtained by mask processing of the multi-modal training data. The method can improve the accuracy of the image classification result obtained by the neural network model.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle driving technology, and in particular to an image classification method, apparatus, computer equipment, storage medium, and program product. Background Technology

[0002] Cameras and radar are two common sensors on autonomous vehicles. Some perception models use the data acquired by both simultaneously as input to classify objects in images after processing. However, autonomous vehicles collect a large amount of data every day, and manually labeling the collected data is inefficient. Therefore, semi-supervised training can be used to train the model.

[0003] In the prior art, computer-aided manufacturing (CAM) is used to generate a background mask for the original data, a foreground image of the original data is generated based on the background mask, and the dataset generated based on the foreground image is input into a deep neural network for semi-supervised training to obtain a trained deep neural network.

[0004] However, the aforementioned techniques suffer from the problem that the trained deep neural networks are susceptible to uncertainty, which in turn affects the accuracy of image classification results obtained through deep neural networks. Summary of the Invention

[0005] This disclosure provides an image classification method, apparatus, computer device, storage medium, and program product, which can be used to improve the accuracy of image classification results performed using a neural network model.

[0006] In a first aspect, embodiments of this disclosure provide an image classification method, the method comprising:

[0007] Acquire multimodal data;

[0008] Determine the target neural network model from multiple pre-trained neural network models;

[0009] The multimodal data is input into the target neural network model for classification processing to determine the target category corresponding to the multimodal data;

[0010] Each of the neural network models is trained on multiple labeled training datasets and multiple unlabeled training datasets. Both the labeled training datasets and the unlabeled training datasets include multimodal training data. The unlabeled training datasets are training datasets after masking the multimodal training data therein.

[0011] Secondly, embodiments of this disclosure provide an image classification apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire multimodal data;

[0013] A determination module is used to determine a target neural network model from multiple pre-trained neural network models; wherein each neural network model is trained based on multiple labeled training datasets and multiple unlabeled training datasets, and both the labeled training datasets and the unlabeled training datasets include multimodal training data, and the unlabeled training datasets are training datasets after masking through the multimodal training data therein;

[0014] The classification module is used to input the multimodal data into the target neural network model for classification processing to determine the target category corresponding to the multimodal data.

[0015] Thirdly, embodiments of this disclosure provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0018] The image classification method, apparatus, computer device, storage medium, and program product provided in this disclosure perform masking processing on unlabeled multimodal training data to obtain an unlabeled training dataset. Multiple neural network models are obtained through semi-supervised training based on multiple labeled training datasets and multiple unlabeled training datasets. The masking processing makes the initial neural network model pay more attention to the foreground of the unlabeled training data during semi-supervised training, reducing the uncertainty of the initial neural network model regarding the unlabeled training dataset and improving the accuracy of the neural network model's image classification results. Furthermore, a target neural network is determined from the multiple neural network models, and the acquired multimodal data is input into the target neural network model for classification processing to determine the target category corresponding to the multimodal data. Selecting the target neural network model makes the obtained classification results more accurate. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the application environment of an image classification method in one embodiment;

[0020] Figure 2This is a flowchart illustrating an image classification method in one embodiment;

[0021] Figure 3 This is a flowchart illustrating an image classification method in another embodiment;

[0022] Figure 4 This is a flowchart illustrating an image classification method in another embodiment;

[0023] Figure 5 This is a flowchart illustrating an image classification method in another embodiment;

[0024] Figure 6 This is a flowchart illustrating an image classification method in another embodiment;

[0025] Figure 7 This is a flowchart illustrating an image classification method in another embodiment;

[0026] Figure 8 This is a flowchart illustrating an image classification method in another embodiment;

[0027] Figure 9 This is a flowchart illustrating an image classification method in another embodiment;

[0028] Figure 10 This is a structural block diagram of an image classification device in one embodiment;

[0029] Figure 11 This is a structural block diagram of an image classification device in another embodiment;

[0030] Figure 12 This is a structural block diagram of an image classification device in another embodiment;

[0031] Figure 13 This is a structural block diagram of an image classification device in another embodiment;

[0032] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this disclosure and are not intended to limit the embodiments of this disclosure.

[0034] First, before introducing the technical solutions of the embodiments of this disclosure in detail, the technical background or evolution of the embodiments of this disclosure will be introduced. Generally, in the field of vehicle driving, the current technical background is as follows: camera sensors and radar sensors are common sensors in autonomous vehicles. Multimodal data collected by camera sensors and radar sensors are used as sample data to train multimodal image classification models. However, multimodal image classification models are sensitive to changes in image illumination. When labeled data is scarce and lacks diversity, training a multimodal image classification model results in an inaccurate classification of images. Furthermore, in the field of image classification, existing technologies utilize computer-aided manufacturing (CAM) to generate background masks for the original data, generate foreground images from the background masks, and input the dataset generated from the foreground images into a deep neural network for semi-supervised training to obtain a trained deep neural network. However, the method of generating background masks using CAM cannot obtain high-quality background masks, thus failing to accurately obtain foreground images and easily misleading the learning of the deep neural network. Therefore, the above methods suffer from the problem that the model is easily affected by uncertainties. How to accurately classify images has become a pressing problem to be solved. In addition, it should be noted that the applicant has put in a great deal of creative effort in identifying the problems of traditional technologies and the technical solutions described in the following embodiments.

[0035] The technical solutions involved in the embodiments of this disclosure will be described below in conjunction with the scenarios in which they are applied.

[0036] The image classification method provided in this disclosure can be applied to, for example... Figure 1 In the application environment shown, computer device 104 communicates with acquisition device 102 via a network. Acquisition device 102 sends the acquired multimodal data to computer device 104. Computer device 104 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0037] In one embodiment, such as Figure 2 As shown, an image classification method is provided, which can be applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0038] S201, acquire multimodal data.

[0039] Multimodal data can include image data, point cloud data, etc.

[0040] In this embodiment, multimodal data of the vehicle during its movement is collected using acquisition devices, which may include cameras, video cameras, scanners, video capture cards, radar, infrared sensors, etc. For example, image data can be collected using a camera, and point cloud data can be collected using radar. The collected image data and point cloud data are used as multimodal data, where the image data can be road images captured by the camera, and the point cloud data corresponds one-to-one with the camera data.

[0041] S202, determine the target neural network model from multiple pre-trained neural network models; wherein each neural network model is trained based on multiple labeled training datasets and multiple unlabeled training datasets, both of which include multimodal training data, and the unlabeled training dataset is a training dataset after masking through the multimodal training data therein.

[0042] In this embodiment, multiple initial neural network models with identical structures but different parameters are constructed. Labeled multimodal training data are used to form multiple labeled training datasets, and unlabeled multimodal training data are used to form multiple unlabeled training datasets. For example, if the multimodal training data includes image data and point cloud data, then a labeled image and its corresponding labeled point cloud are used as a single labeled dataset. The unlabeled image and point cloud data are then masked, and the masked unlabeled image and its corresponding unlabeled point cloud are used as a single unlabeled training dataset. Furthermore, based on the multiple labeled and unlabeled training datasets, the multiple initial neural network models are trained to obtain multiple trained neural network models.

[0043] Optionally, any one of the multiple neural network models can be selected as the target neural network model; or, the same multimodal data can be input into multiple neural network models respectively, and the neural network model with the most accurate image classification result for the multimodal data can be selected as the target neural network model.

[0044] S203, input the multimodal data into the target neural network model for classification processing to determine the target category corresponding to the multimodal data.

[0045] The target categories can include people and things. People can include adults and minors, and things can include animals, plants, and objects.

[0046] In this embodiment, multimodal data is input into a target neural network model. The target neural network classifies the multimodal data and outputs probability values ​​for each category. The category corresponding to the highest probability value among all categories is determined as the target category. For example, if the multimodal data is input into the target neural network model, and the model outputs a probability value of 0.8 for the multimodal data as a human and a probability value of 0.2 for the multimodal data as an object, then the target category corresponding to the multimodal data is a human.

[0047] It should be noted that when multimodal data includes multiple identification objects, it may affect the accuracy of classification of the multimodal data. Identification objects include the primary object and secondary objects. The target category corresponding to the multimodal data is the target category corresponding to the primary object. For example, when the probability value of the multimodal data being a person is 0.8 and the probability value of the multimodal data being an object is 0.2, the target category corresponding to the multimodal data is a person, and the probability value of the multimodal data being an object may correspond to a secondary object at the edge of the multimodal data.

[0048] In the aforementioned image classification method, unlabeled multimodal training data is masked to obtain an unlabeled training dataset. Multiple neural network models are then obtained through semi-supervised training based on multiple labeled and unlabeled training datasets. The masking process makes the initial neural network model pay more attention to the foreground of the unlabeled training data during semi-supervised training, reducing the uncertainty of the initial neural network model regarding the unlabeled training dataset and improving the accuracy of the neural network model's image classification results. Furthermore, a target neural network is determined from the multiple neural network models, and the acquired multimodal data is input into the target neural network model for classification processing to determine the target category corresponding to the multimodal data. Selecting the target neural network model makes the obtained classification results more accurate.

[0049] The training process of the neural network model will be described in detail below. In one embodiment, the unlabeled training dataset includes first modality training data and second modality training data, such as... Figure 3 As shown, the training process of a neural network model includes:

[0050] S301, perform masking processing on the second modality training data based on the first modality training data, and determine the masked second modality training data.

[0051] Optionally, the first modality training data and the second modality training data can be image data with the same geometric position. Using photosensitive conditions and photographic processing, the first modality training data and the second modality training data are processed into image data with different densities and different contrasts. Then, the first modality training data is superimposed on the second modality training data to obtain the masked second modality training data.

[0052] Optionally, in this embodiment, the first modality training data can be point cloud data acquired by radar, and the second modality training data can be image data acquired by a camera. The first and second modality training data correspond one-to-one. The first modality training data is projected onto the second modality training data, and a foreground mask extraction process is performed on the second modality training data to obtain the masked second modality training data. For example, the first modality training data acquired by radar is projected onto the second modality training data acquired by the camera. An AND / OR operation is performed on the second modality training data based on the first modality training data to extract the foreground mask image from the second modality training data, which is the masked second modality training data.

[0053] S302, based on the masked second modality training data and the corresponding first modality training data, determine each unlabeled training dataset.

[0054] In this embodiment, the masked second modality training data is matched one-to-one with the first modality training data, and the masked second modality training data and the corresponding first modality training data are combined into a data pair as an unlabeled training dataset, thereby determining each unlabeled training dataset.

[0055] S303, train each initial neural network model based on each labeled training dataset and each unlabeled training dataset to obtain each neural network model.

[0056] In this embodiment, each initial neural network model is a neural network model with the same structure but different parameters. Semi-supervised training is performed on each initial neural network model using both labeled and unlabeled training datasets to obtain the final neural network model. Optionally, during each training iteration, a portion of the datasets from both labeled and unlabeled training datasets is selected as the training dataset according to a preset ratio.

[0057] Optionally, training of each initial neural network model can be completed when a preset number of training iterations are reached; or, training of each initial neural network model can be completed when the loss function value of each initial neural network is less than a preset threshold.

[0058] Optionally, the initial neural network model in this embodiment can be a model such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Deep Neural Network (DNN), or a Fully Convolutional Network (FCN).

[0059] In this embodiment, semi-supervised training is performed on each initial neural network model using labeled and unlabeled training sets. The unlabeled training dataset is obtained by masking the second modality training data based on the first modality training data. The unlabeled training dataset is determined based on the first modality training data and the masked second modality training data. The masking process makes the objects to be identified in the second modality training data more prominent, thereby improving the training effect of the model and making the classification results of the obtained neural network model more accurate for multimodal data.

[0060] In the scenario described above, where each initial neural network model is trained using both labeled and unlabeled training datasets to obtain the final neural network model, the labeled and standard training datasets are first augmented, and then the initial neural network models are trained using the augmented labeled and unlabeled training datasets. In one embodiment, such as... Figure 4 As shown, the above S303 includes:

[0061] S401 performs augmentation processing on each labeled training dataset and each unlabeled training dataset.

[0062] The enhancement processing may include random cropping, random flipping, scaling, random color dithering, random brightness adjustment, and random Gaussian noise processing.

[0063] In this embodiment, each labeled training dataset is augmented to obtain augmented labeled training datasets; each unlabeled training dataset is augmented to obtain augmented unlabeled training datasets.

[0064] Optionally, one or more processing methods can be selected from random cropping, random flipping, scaling, random color jittering, random brightness adjustment, and random Gaussian noise processing to process each labeled training dataset and each unlabeled training dataset.

[0065] S402, each initial neural network model is trained using the enhanced labeled training dataset and the enhanced unlabeled training dataset to obtain each neural network model.

[0066] Optionally, the enhanced labeled training datasets and the enhanced unlabeled training datasets can be input into each initial neural network model for semi-supervised training. The loss value is calculated based on the prediction results of each initial neural network model on each labeled training dataset and the prediction results on each unlabeled training dataset. The parameters of each neural network model are then adjusted based on the loss value to obtain each neural network model.

[0067] In this embodiment, the labeled training datasets and the unlabeled training datasets are enhanced to make the features of each dataset more prominent. Therefore, the enhanced labeled training datasets and the enhanced unlabeled training datasets are used to train each initial neural network model, resulting in better classification performance of each neural network.

[0068] The following section will describe in detail the process of training each initial neural network model using both the enhanced labeled training datasets and the enhanced unlabeled training datasets to obtain the final neural network model. In one embodiment, such as... Figure 5 As shown, the above S402 includes:

[0069] S501, the enhanced unlabeled training datasets are input into the initial teacher model to determine the first prediction result corresponding to each unlabeled training dataset.

[0070] In this embodiment, the unlabeled data undergoes weak augmentation processing, which includes geometric transformations such as random cropping, random flipping, and scaling, as well as color and noise transformations. The weakly augmented unlabeled training data is then input into the initial teacher model, which classifies the weakly augmented unlabeled training data to obtain the first probability value of each category corresponding to the weakly augmented unlabeled data. The first probability value of each category is then used as the first prediction result corresponding to the unlabeled training dataset.

[0071] S502, the enhanced labeled training datasets and the enhanced unlabeled training datasets are input into the initial student model to obtain the second prediction result for each labeled training dataset and the third prediction result for each unlabeled training dataset.

[0072] In this embodiment, the labeled data undergoes strong enhancement processing, including geometric transformations, color transformations, and noise transformations such as random cropping, random flipping, scaling, random color jittering, random brightness adjustment, and random Gaussian noise processing. The enhanced labeled training dataset is then input into an initial student model, which classifies the enhanced labeled training data to obtain the second probability value for each category. This second probability value is used as the second prediction result for the labeled training dataset. Similarly, the unlabeled data undergoes strong enhancement processing. The enhanced unlabeled training dataset is then input into the initial student model, which classifies the enhanced unlabeled training data to obtain the third probability value for each category. This third probability value is used as the third prediction result for the unlabeled training dataset.

[0073] S503, the initial student model and the initial teacher model are trained based on the first prediction results, the second prediction results and the third prediction results to obtain the teacher model and the student model.

[0074] In this embodiment, the parameters of the initial student model and the initial teacher model are optimized based on the first prediction result, the second prediction result, and the third prediction result to obtain the teacher model and the student model. Optionally, the loss values ​​of the first prediction result and the second prediction result, and the loss values ​​of the second prediction result and the third prediction result can be calculated respectively. Further, the parameters of the student model are optimized based on the loss values ​​of the first prediction result and the second prediction result, and the parameters of the teacher model are optimized based on the loss values ​​of the second prediction result and the third prediction result, thereby obtaining the teacher model and the student model.

[0075] In this embodiment, based on the enhanced unlabeled training data, the enhanced labeled training data, the initial learning model, and the initial teacher model, a first prediction result, a second prediction result, and a third prediction result are obtained. The initial learning model and the initial teacher model are trained using each prediction result, resulting in a teacher model and a student model with higher classification accuracy.

[0076] The following section details the process of training the initial student model and the initial teacher model based on the first, second, and third prediction results to obtain the teacher and student models. In one embodiment, each labeled training dataset includes multimodal training data and the corresponding label categories for each multimodal training data, such as... Figure 6 As shown, the above S503 includes:

[0077] S601, calculate the first loss between each first prediction result and the corresponding third prediction result.

[0078] In this embodiment, a consistency loss value between the first prediction result and the corresponding third prediction result is calculated, and this consistency loss value is used as the first loss. The smaller the first loss, the smaller the difference between the first prediction result and the third prediction result; the larger the first loss, the larger the difference between the first prediction result and the third prediction result.

[0079] Optionally, the first loss between each first prediction result and the corresponding third prediction result can be calculated using the mean squared error loss function.

[0080] S602, calculate the second loss between each second prediction result and the corresponding labeled category.

[0081] In this embodiment, the classification loss value between the second prediction result and the corresponding label category is calculated, and this classification loss value is used as the second loss. The smaller the second loss, the smaller the difference between the second prediction result and the corresponding label category; the larger the second loss, the larger the difference between the second prediction result and the corresponding label category.

[0082] Optionally, a second loss can be calculated between each second prediction result and the corresponding labeled category using the cross-entropy loss function.

[0083] S603, train the initial student model and the initial teacher model based on the first loss and the second loss to obtain the teacher model and the student model.

[0084] Optionally, the parameters of the teacher model can be optimized based on the first loss to minimize the first loss, and the parameters of the student model can be optimized based on the second loss to minimize the second loss; or, the parameters of the student model can be optimized based on the first loss and the second loss, and the parameters of the teacher model can be optimized based on the first loss.

[0085] In this embodiment, a first loss and a second loss are calculated based on each prediction result, thereby optimizing the initial teacher model and the initial student model using the first loss and the second loss. Using two losses to optimize the model makes the training efficiency of the initial teacher model and the initial student model higher, and the classification accuracy of the trained teacher model and student model is higher.

[0086] The following will elaborate on the process of training the initial student model and the initial teacher model based on the first loss and the second loss to obtain the teacher model and the student model. In one embodiment, as follows... Figure 7 As shown, the above S603 includes:

[0087] S701, optimize the parameters of the initial student model based on the first loss and the second loss to obtain the optimized student model.

[0088] In this embodiment, the first loss and the second loss are weighted and summed according to the preset first coefficient and the preset second coefficient to obtain the target loss. The parameters of the student model are then optimized according to the target loss to obtain the optimized student model.

[0089] Optionally, the first coefficient can be 5, and the second coefficient can increase linearly between 0 and 10. When the number of training iterations exceeds a preset number, the second coefficient is set to 10. For example, if the first loss is 'a' and the second loss is 'b', with the first coefficient being 5 and the second coefficient being 10, the target loss is...

[0090]

[0091] S702, optimize the parameters of the initial teacher model based on the optimized student model to obtain the optimized teacher model.

[0092] In this embodiment, based on the parameters of the optimized student model, the corresponding parameters in the teacher model are optimized using the Exponential Moving Average (EMA) metric to obtain the optimized teacher model. For example, when using EMA to optimize the parameters of the initial teacher model, the proportion of parameters in the optimized learning model is 0.001, and the proportion of parameters in the initial teacher model is 0.999.

[0093] S703, determine the teacher model and student model based on the optimized student model and the optimized teacher model.

[0094] In this embodiment, the optimized teacher model is used as the initial teacher model for the next round of training, and the optimized student model is used as the initial student model for the next round of training. When the number of training sessions reaches the preset number of training sessions, the training is completed, and the current optimized teacher model and optimized student model are determined as the teacher model and student model, respectively.

[0095] In this embodiment, the parameters of the initial student model are optimized based on the first loss and the second loss, and then the parameters of the initial teacher model are optimized based on the optimized student model. Compared with optimizing the parameters of the initial teacher model and the initial student model simultaneously based on the first loss and the second loss, the method provided in this embodiment is more accurate in optimizing the parameters of the initial teacher model.

[0096] In the scenario described above, where the first loss between each first prediction result and the corresponding third prediction result is calculated, the first prediction results are first filtered, and then the first loss between the filtered target prediction result and the corresponding third prediction result is calculated. In one embodiment, such as Figure 8 As shown, the above S601 includes:

[0097] S801, the first prediction results are filtered according to the preset threshold conditions to determine the prediction results of each target.

[0098] In this embodiment, the first prediction result is a set of classification results obtained by the initial teacher model based on the enhanced unlabeled training data. Each set of classification results includes the probability value of each enhanced unlabeled training data belonging to each type. If the probability value in the classification result is less than the threshold condition, the probability of the enhanced unlabeled training data belonging to each type is low. That is, the classification accuracy of the initial teacher model for the enhanced unlabeled training data is low. Therefore, the classification result corresponding to the enhanced unlabeled training data in the first prediction result is removed, and the remaining classification results in the first prediction result are used as the prediction results for each target.

[0099] For example, if the first prediction result includes classification result 1, classification result 2 and classification result 3, the threshold condition is 0.8, the probability value of pedestrian in classification result 1 is 0.9 and the probability value of vehicle is 0.1, the probability value of pedestrian in classification result 2 is 0.2 and the probability value of vehicle is 0.8, and the probability value of pedestrian in classification result 3 is 0.6 and the probability value of vehicle is 0.4, then all the probability values ​​in classification result 3 are less than the threshold condition. Therefore, classification result 3 in the first prediction result is removed, and classification result 1 and classification result 2 are used as the target prediction results.

[0100] S802, calculate the first loss between each target prediction result and the corresponding third prediction result.

[0101] In this embodiment, the third prediction result is the multiple classification results obtained by the initial student model based on the enhanced unlabeled training data. The classification result corresponding to each target prediction result is determined in the third prediction result. The consistency loss value between each target prediction result and the corresponding third prediction result is calculated. The consistency loss value is used as the first loss between each target prediction result and the corresponding third prediction result.

[0102] In this embodiment, each first prediction result is filtered according to a preset threshold condition, and inaccurate classification results in the first prediction results are removed to obtain the target prediction result, thereby making the accuracy of the first loss calculated based on the target prediction result and the response third prediction result higher.

[0103] The following describes an embodiment of this disclosure using a specific image classification scenario. See details below. Figure 9 As shown, the method includes the following steps:

[0104] S1, Project the first modality training data onto the second modality training data, and perform foreground mask extraction processing on the second modality training data to obtain the masked second modality training data.

[0105] S2, based on the masked second modality training data and the corresponding first modality training data, determine each unlabeled training dataset.

[0106] S3 performs augmentation processing on both labeled and unlabeled training datasets.

[0107] S4. Input the enhanced unlabeled training datasets into the initial teacher model to determine the first prediction result corresponding to each unlabeled training dataset; input the enhanced labeled training datasets and the enhanced unlabeled training datasets into the initial student model to obtain the second prediction result of each labeled training dataset and the third prediction result of each unlabeled training dataset.

[0108] S5, filter each first prediction result according to the preset threshold conditions, determine each target prediction result, calculate the first loss between each target prediction result and the corresponding third prediction result; calculate the second loss between each second prediction result and the corresponding labeling category.

[0109] S6. Optimize the parameters of the initial student model based on the first loss and the second loss to obtain the optimized student model.

[0110] S7. Optimize the parameters of the initial teacher model based on the optimized student model to obtain the optimized teacher model.

[0111] S8. Determine the teacher model and student model based on the optimized student model and the optimized teacher model.

[0112] S9. Acquire multimodal data; determine the target neural network model from multiple pre-trained neural network models; input the multimodal data into the target neural network model for classification processing, and determine the target category corresponding to the multimodal data.

[0113] In the aforementioned image classification method, unlabeled multimodal training data is masked to obtain an unlabeled training dataset. Multiple neural network models are then obtained through semi-supervised training based on multiple labeled and unlabeled training datasets. The masking process makes the initial neural network model pay more attention to the foreground of the unlabeled training data during semi-supervised training, reducing the uncertainty of the initial neural network model regarding the unlabeled training dataset and improving the accuracy of the neural network model's image classification results. Furthermore, a target neural network is determined from the multiple neural network models, and the acquired multimodal data is input into the target neural network model for classification processing to determine the target category corresponding to the multimodal data. Selecting the target neural network model makes the obtained classification results more accurate.

[0114] It should be understood that, although Figure 2-9 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-9 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0115] In one embodiment, such as Figure 10 As shown, an image classification device is provided, including: an acquisition module 10, a first determination module 11, and a classification module 12, wherein:

[0116] Acquisition module 10 is used to acquire multimodal data.

[0117] The first determining module 11 is used to determine the target neural network model from multiple pre-trained neural network models; wherein each neural network model is trained based on multiple labeled training datasets and multiple unlabeled training datasets, both of which include multimodal training data, and the unlabeled training dataset is a training dataset after masking through the multimodal training data therein.

[0118] The classification module 12 is used to input multimodal data into the target neural network model for classification processing and to determine the target category corresponding to the multimodal data.

[0119] The image classification device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0120] In one embodiment, such as Figure 11 As shown, the above-mentioned device further includes: a processing module 13, a second determining module 14, and a training module 15, wherein:

[0121] The processing module 13 is used to perform masking processing on the second modality training data based on the first modality training data, and to determine the masked second modality training data.

[0122] The second determining module 14 is used to determine each unlabeled training dataset based on the masked second modality training data and the corresponding first modality training data.

[0123] Training module 15 is used to train each initial neural network model based on each labeled training dataset and each unlabeled training dataset to obtain each neural network model.

[0124] The image classification device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0125] In one embodiment, such as Figure 12 As shown, the above-mentioned processing module 13 includes a first processing unit 131, which is used to project the first modality training data onto the second modality training data, and to perform foreground mask extraction processing on the second modality training data to obtain the masked second modality training data.

[0126] The image classification device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0127] In one embodiment, such as Figure 13 As shown, the training module 15 includes: a second processing unit 151 and a training unit 152, wherein:

[0128] The second processing unit 151 is used to perform augmentation processing on each labeled training dataset and each unlabeled training dataset.

[0129] Training unit 152 is used to train each initial neural network model using the enhanced labeled training datasets and the enhanced unlabeled training datasets to obtain each neural network model.

[0130] The image classification device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0131] In one embodiment, the training unit 152 is configured to input the enhanced unlabeled training datasets into the initial teacher model to determine the first prediction result corresponding to each unlabeled training dataset; input the enhanced labeled training datasets and the enhanced unlabeled training datasets into the initial student model to obtain the second prediction result of each labeled training dataset and the third prediction result of each unlabeled training dataset; and train the initial student model and the initial teacher model based on the first prediction result, the second prediction result, and the third prediction result to obtain the teacher model and the student model.

[0132] The image classification device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0133] In one embodiment, the training unit 152 is used to calculate a first loss between each first prediction result and the corresponding third prediction result; calculate a second loss between each second prediction result and the corresponding labeled category; and train an initial student model and an initial teacher model based on the first loss and the second loss to obtain a teacher model and a student model.

[0134] The image classification device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0135] In one embodiment, the training unit 152 is configured to optimize the parameters of an initial student model based on a first loss and a second loss to obtain an optimized student model; optimize the parameters of an initial teacher model based on the optimized student model to obtain an optimized teacher model; and determine a teacher model and a student model based on the optimized student model and the optimized teacher model.

[0136] The image classification device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0137] In one embodiment, the training unit 152 is used to filter each first prediction result according to a preset threshold condition to determine each target prediction result; and to calculate the first loss between each target prediction result and the corresponding third prediction result.

[0138] The image classification device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0139] For specific limitations regarding the image classification device, please refer to the limitations on the image classification method above, which will not be repeated here. Each module in the aforementioned image classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the electronic device in hardware form, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0140] Figure 14 This is a block diagram illustrating a computer device 1400 according to an exemplary embodiment. (Refer to...) Figure 14The computer device 1400 includes a processing component 1420, which further includes one or more processors, and memory resources represented by memory 1422 for storing instructions or computer programs, such as application programs, that can be executed by the processing component 1420. The application programs stored in memory 1422 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1420 is configured to execute instructions to perform the image classification method described above.

[0141] Computer device 1400 may also include a power supply component 1424 configured to perform power management of device 1400, a wired or wireless network interface 1426 configured to connect device 1400 to a network, and an input / output (I / O) interface 1428. Computer device 1400 can operate on an operating system stored in memory 1422, such as Windows 1414erver™, Mac OS X™, Unix™, Linux™, FreeB14D™, or similar.

[0142] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 1422 including instructions, which can be executed by a processor of a computer device 1400 to perform the above-described method. The storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0143] In an exemplary embodiment, a computer program product is also provided, which, when executed by a processor, can implement the above-described methods. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, some or all of the above-described methods can be implemented, wholly or partially, according to the processes or functions described in the embodiments of this disclosure.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-described embodiments are merely illustrative of several implementation methods of the present disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present disclosure embodiments, and these all fall within the protection scope of the present disclosure embodiments. Therefore, the protection scope of the patent for the embodiments of the present disclosure should be determined by the appended claims.

Claims

1. An image classification method, characterized in that, The method includes: Acquire multimodal data; the multimodal data includes image data and point cloud data; Determine the target neural network model from multiple pre-trained neural network models; The multimodal data is input into the target neural network model for classification processing to determine the target category corresponding to the multimodal data; Each of the aforementioned neural network models includes a teacher model and a student model. Each neural network model is obtained by inputting multiple enhanced unlabeled training datasets into an initial teacher model, and inputting the multiple enhanced labeled training datasets and the multiple enhanced unlabeled training datasets into an initial student model, and training the initial teacher model and the initial student model. Both the labeled training dataset and the unlabeled training dataset include multimodal training data. The unlabeled training dataset is a training dataset after masking the multimodal training data therein. The unlabeled training dataset includes first modality training data and second modality training data. The masked training dataset is obtained by masking the second modality training data based on the first modality training data. The first modality training data is point cloud data, and the second modality training data is image data.

2. The method according to claim 1, characterized in that, The training process for each of the aforementioned neural network models includes: The second modality training data is masked based on the first modality training data to determine the masked second modality training data. Based on the masked second modality training data and the corresponding first modality training data, each of the unlabeled training datasets is determined; Augmentation processing is performed on each of the labeled training datasets and each of the unlabeled training datasets; Multiple augmented labeled training sets and multiple augmented unlabeled training datasets are input into the initial student model to train the initial teacher model and the initial student model, thereby obtaining each neural network model.

3. The method according to claim 2, characterized in that, The step of performing masking processing on the second modality training data based on the first modality training data to determine the masked second modality training data includes: The first modality training data is projected onto the second modality training data, and the second modality training data is subjected to foreground mask extraction processing to obtain the masked second modality training data.

4. The method according to claim 2, characterized in that, The step of inputting multiple augmented labeled training sets and multiple augmented unlabeled training datasets into the initial student model, and training the initial teacher model and the initial student model to obtain each of the neural network models includes: The enhanced unlabeled training datasets are input into the initial teacher model to determine the first prediction result corresponding to each unlabeled training dataset. The enhanced labeled training datasets and the enhanced unlabeled training datasets are input into the initial student model to obtain the second prediction result of each labeled training dataset and the third prediction result of each unlabeled training dataset. The initial student model and the initial teacher model are trained based on the first prediction result, the second prediction result, and the third prediction result to obtain the teacher model and the student model.

5. The method according to claim 4, characterized in that, Each of the labeled training datasets includes multimodal training data and labeled categories corresponding to each of the multimodal training data. The step of training the initial student model and the initial teacher model based on each of the first prediction results, each of the second prediction results, and each of the third prediction results to obtain the teacher model and the student model includes: Calculate the first loss between each of the first prediction results and the corresponding third prediction results; Calculate the second loss between each second prediction result and the corresponding labeled category; The initial student model and the initial teacher model are trained based on the first loss and the second loss to obtain the teacher model and the student model.

6. The method according to claim 5, characterized in that, The step of training the initial student model and the initial teacher model based on the first loss and the second loss to obtain the teacher model and the student model includes: The parameters of the initial student model are optimized based on the first loss and the second loss to obtain the optimized student model. Based on the optimized student model, the parameters of the initial teacher model are optimized to obtain the optimized teacher model; The teacher model and the student model are determined based on the optimized student model and the optimized teacher model.

7. The method according to claim 5, characterized in that, The calculation of the first loss between each of the first prediction results and the corresponding third prediction results includes: The first prediction results are filtered according to preset threshold conditions to determine the prediction results of each target. Calculate the first loss between each of the target prediction results and the corresponding third prediction results.

8. An image classification device, characterized in that, The device includes: An acquisition module is used to acquire multimodal data; the multimodal data includes image data and point cloud data; A first determining module is used to determine a target neural network model from a plurality of pre-trained neural network models. Each neural network model includes a teacher model and a student model. Each neural network model is obtained by inputting a plurality of augmented unlabeled training datasets into an initial teacher model, and inputting a plurality of augmented labeled training datasets and the plurality of augmented unlabeled training datasets into an initial student model, and training the initial teacher model and the initial student model. Both the labeled training dataset and the unlabeled training dataset include multimodal training data. The unlabeled training dataset is a training dataset after masking the multimodal training data. The unlabeled training dataset includes first modality training data and second modality training data. The masked training dataset is obtained by masking the second modality training data based on the first modality training data. The first modality training data is point cloud data, and the second modality training data is image data. The classification module is used to input the multimodal data into the target neural network model for classification processing to determine the target category corresponding to the multimodal data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images