A training method for image classification and generation models in scenarios with incomplete data.

Abstract

This invention discloses a training method for image classification and generation models in scenarios with incomplete data. The method involves acquiring the labels and image information of the original image; processing the image information using a VQ-VAE to obtain first discrete information; inputting the labels and image information into a data processing model to obtain second discrete information; constructing an initial Big Learning model based on the second discrete information; and training the initial Big Learning model based on the first discrete information to obtain a final Big Learning model. By using the first discrete information obtained through VQ-VAE processing and the second discrete information obtained through the data processing model, a Big Learning model that accurately identifies objects and features in images can be trained even in scenarios with missing or incomplete data. This improves the accuracy of image classification and generation and can be widely applied in the field of computer data processing technology.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a training method for image classification and generation models in scenarios with incomplete data. Background Technology

[0002] In computer vision, image classification and image generation are two key tasks. Image classification aims to identify subjects, objects, or specific features in an image and categorize them into predefined classes, while image generation aims to generate new images by learning the inherent structure and distribution of existing data. However, in real-world applications, the available data is not always complete due to factors such as difficulties in data collection, sensor malfunctions, and privacy concerns.

[0003] Currently, existing image classification and generation models require a large amount of complete data to learn accurate features and patterns during the training process. When data is missing or incomplete, it will affect the training model's ability to recognize objects and features in images, thereby reducing the accuracy of image classification and generation. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a training method for image classification and generation models in scenarios with incomplete data, which can train a model that accurately identifies objects and features in images in scenarios with missing or incomplete data, thereby improving the accuracy of image classification and generation.

[0005] In a first aspect, embodiments of the present invention provide a method for training image classification and generation models in scenarios with incomplete data, comprising:

[0006] Obtain the label and image information of the original image;

[0007] The image information is input into VQ-VAE for processing to obtain the first discrete information. The VQ-VAE is used to convert the image information into encoded information.

[0008] After inputting the label and the image information into the data processing model, second discrete information is obtained. The data processing model is used to encode the label and the image information respectively to obtain label code and image code. The second discrete information includes the label code and the image code.

[0009] An initial Big Learning model is constructed based on the second discrete information. The initial Big Learning model represents an image classification and generation model. The second discrete information is masked by a random masking rate to construct the initial Big Learning model.

[0010] The initial Big Learning model is trained based on the first discrete information to obtain the final Big Learning model.

[0011] Optionally, constructing the initial Big Learning model based on the second discrete information specifically includes:

[0012] Target data and source data are obtained based on the second discrete information and the random masking rate, wherein the target data represents the masked data and the source data represents the unmasked data;

[0013] A network architecture for permutation language modeling was constructed based on BERT and XLNET.

[0014] A pre-trained objective function is obtained based on the permutation language modeling network architecture, the target data, and the source data;

[0015] The initial BigLearning model is determined based on the permutation language modeling network architecture and the pre-training objective function.

[0016] Optionally, obtaining the target data and source data based on the second discrete information and the random occlusion rate specifically includes:

[0017] Obtain the random occlusion rate;

[0018] The target data and the source data are obtained by masking the label encoding or the image encoding according to the random masking rate.

[0019] Optionally, after obtaining the target data and the source data by masking the label encoding or the image encoding according to the random masking rate, the method further includes:

[0020] If the target data is the label encoding and the source data is the image encoding, then the image classification task is started;

[0021] If the target data is the image encoding and the source data is the tag encoding, then the image condition generation task is started;

[0022] If the target data is the image encoding and the source data is an empty set, then the unconditional image generation task is started.

[0023] Optionally, the construction of the permutation language modeling network architecture based on BERT and XLNET specifically includes:

[0024] A masking module is constructed based on the BERT, which is used to input and output masks to obtain different source data and target data.

[0025] An arrangement module is constructed based on the XLNET, which is used to arrange the encoding of the target data;

[0026] The permutation language modeling network architecture is obtained based on the permutation module and the mask module.

[0027] Optionally, the step of inputting the image information into a VQ-VAE for processing to obtain the first discrete information specifically includes:

[0028] The image information is input into the encoder of the VQ-VAE to obtain continuous encoded information;

[0029] The first discrete information is obtained by mapping the continuous encoded information to the encoding cluster of the VQ-VAE.

[0030] Optionally, the step of inputting the label and the image information into the data processing model to obtain the second discrete information specifically includes:

[0031] The image information is input into the data processing model for image modality encoding to obtain the image code;

[0032] The label is input into the data processing model and subjected to label modal encoding to obtain the label code;

[0033] The second discrete information is obtained by concatenating the image encoding and the label encoding.

[0034] Secondly, embodiments of the present invention provide a training system for image classification and generation models in scenarios with incomplete data, comprising:

[0035] The first module is used to obtain the labels and image information of the original image;

[0036] The second module is used to input the image information into VQ-VAE for processing to obtain the first discrete information, wherein the VQ-VAE is used to convert the image information into encoded information;

[0037] The third module is used to input the label and the image information into the data processing model to obtain the second discrete information. The data processing model is used to encode the label and the image information respectively to obtain the label code and the image code. The second discrete information includes the label code and the image code.

[0038] The fourth module is used to construct an initial Big Learning model based on the second discrete information. The initial Big Learning model represents an image classification and generation model. The second discrete information is used to construct the initial Big Learning model after being masked by a random masking rate.

[0039] The fifth module is used to train the initial Big Learning model based on the first discrete information to obtain the final Big Learning model.

[0040] Thirdly, embodiments of the present invention provide a training apparatus for image classification and generation models in scenarios with incomplete data, comprising:

[0041] At least one processor;

[0042] At least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor performs the method as described above.

[0044] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, characterized in that the processor-executable program, when executed by a processor, is used to perform the method described above.

[0045] Implementing the embodiments of the present invention has the following beneficial effects: A training method for an image classification and generation model in a data-incomplete scenario, as described in this embodiment, includes: acquiring the label and image information of the original image; inputting the image information into a VQ-VAE for processing to obtain first discrete information, wherein the VQ-VAE is used to convert the image information into encoded information; inputting the label and the image information into a data processing model to obtain second discrete information, wherein the data processing model is used to encode the label and the image information respectively to obtain label encoding and image encoding, wherein the second discrete information includes the label encoding and the image encoding; constructing an initial Big Learning model based on the second discrete information, wherein the initial Big Learning model represents the image classification and generation model, wherein the second discrete information is masked using a random occlusion rate to construct the initial Big Learning model; and training the initial Big Learning model based on the first discrete information to obtain a final Big Learning model. The first discrete information is obtained through VQ-VAE processing, and the second discrete information obtained through the data processing model is used to construct an initial Big Learning model. The initial Big Learning model is then trained using the first discrete information to obtain the final Big Learning model. This method can train a Big Learning model that accurately identifies objects and features in images even in scenarios with missing or incomplete data, thereby improving the accuracy of image classification and generation. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the steps in training a model for image classification and generation in a data-incomplete scenario, as provided in an embodiment of the present invention.

[0047] Figure 2 This is a flowchart of the process for constructing a Big Learning model provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the VQ-VAE model provided in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the data processing model provided in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the Big Learning model provided in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the permutation language modeling network architecture provided in an embodiment of the present invention;

[0052] Figure 7 This is an example diagram of an image classification task provided in an embodiment of the present invention;

[0053] Figure 8 This is an example image of image generation provided in an embodiment of the present invention;

[0054] Figure 9 This is an example diagram showing the results of an image classification task provided in an embodiment of the present invention;

[0055] Figure 10 This is a heatmap of image classification provided in an embodiment of the present invention;

[0056] Figure 11 This is a schematic diagram of the MSTAR classification results provided in an embodiment of the present invention;

[0057] Figure 12 This is an example image of unconditional image generation using the SAR dataset provided in this embodiment of the invention;

[0058] Figure 13 (a) is an example diagram of conditional generation under the MNIST dataset provided in the embodiments of the present invention;

[0059] Figure 13 (b) is an example image unconditionally generated using the MNIST dataset provided in this embodiment of the invention;

[0060] Figure 14 (a) is an example image of image completion on the MNIST dataset provided in an embodiment of the present invention;

[0061] Figure 14(b) is an example image of image completion in the CelebA dataset provided in this embodiment of the invention;

[0062] Figure 15 This is a structural block diagram of a training system for an image classification and generation model in a data-incomplete scenario, provided by an embodiment of the present invention.

[0063] Figure 16 This is a structural block diagram of a training device for an image classification and generation model in a data-incomplete scenario, provided by an embodiment of the present invention. Detailed Implementation

[0064] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0065] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0066] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0067] In the description of this invention, unless otherwise explicitly defined, terms such as "setting," "installing," and "connecting" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0068] like Figure 1 As shown in the figure, this embodiment of the invention provides a training method for image classification and generation models in scenarios with incomplete data, which includes the following steps.

[0069] S100: Obtain the label and image information of the original image.

[0070] Specifically, the collected raw images are processed to obtain the corresponding labels and image information, and the labels and image information are matched with each other.

[0071] S200. The image information is input into VQ-VAE for processing to obtain first discrete information. The VQ-VAE is used to convert the image information into encoded information.

[0072] Reference Figure 2-3 Specifically, the acquired image information is input into a VQ-VAE encoder, which is a deep neural network. The VQ-VAE encoder converts the input image information (typically RGB pixel values) into a set of low-dimensional continuous codes, called the latent space. The continuous codes output by the VQ-VAE encoder are mapped to a pre-defined discrete coding space. This pre-defined discrete coding space typically consists of a set of fixed vectors (called coding clusters or coding dictionaries). Each continuous code is mapped to the nearest coding cluster, thus quantizing the continuous code into a discrete form, thereby obtaining the first discrete information.

[0073] Optionally, the step of inputting the image information into a VQ-VAE for processing to obtain the first discrete information specifically includes:

[0074] S210. Input the image information into the encoder of the VQ-VAE to obtain continuous encoding information;

[0075] S220. Map the continuous encoded information to the encoding cluster of the VQ-VAE to obtain the first discrete information.

[0076] Reference Figure 3 Specifically, after the image information is input into the VQ-VAE encoder, the VQ-VAE encoder can convert the input image information into continuous encoded information, that is, to encode the image information one by one to obtain continuous encoded information.

[0077] The VQ-VAE model also includes a VQ-VAE decoder. Similar to the encoder, the VQ-VAE decoder is also a deep neural network. The VQ-VAE decoder transforms the discrete encoding (first discrete information) back into the original image space to obtain the reconstructed image. The decoder's goal is to minimize the difference between the reconstructed image and the original image (i.e., the image information input to the VQ-VAE encoder). This is achieved through continuous training of the VQ-VAE model.

[0078] VQ-VAE Training and Optimization: The training objective of VQ-VAE is to minimize reconstruction error and encoding error. Reconstruction error is generated by the decoder, representing the difference between the reconstructed image and the original image; encoding error is the difference between consecutive codes and the coding clusters they are mapped to. By iteratively optimizing these two errors, VQ-VAE can learn better encoders, decoders, and encoding dictionaries.

[0079] S300. The label and the image information are input into the data processing model to obtain the second discrete information. The data processing model is used to encode the label and the image information respectively to obtain the label code and the image code. The second discrete information includes the label code and the image code.

[0080] Reference Figure 4 Specifically, the label and image information are input into the data processing model for processing. The data processing model encodes the labels to obtain label codes, and the data processing model encodes the image information to obtain image codes.

[0081] Optionally, the step of inputting the label and the image information into the data processing model to obtain the second discrete information specifically includes:

[0082] S310. The image information is input into the data processing model to perform image modality encoding to obtain the image encoding;

[0083] S320. The label is input into the data processing model for label modal encoding to obtain the label code;

[0084] S330. The second discrete information is obtained by concatenating the image encoding and the tag encoding.

[0085] Specifically, refer to Figure 4 Image modality coding (IMC) involves segmenting an input image (i.e., the image information of the input data processing model) into a set of fixed-size sub-images, commonly referred to as patches. For each patch, a convolutional neural network (CNN) is used to convert it into a feature vector. The CNN extracts local and global features of the image through multiple convolutional and pooling layers. The resulting image feature vector retains a large amount of information from the original image, thus providing effective image information for subsequent training. The obtained feature vector (I1-I9) is the image code.

[0086] The acquired image labels are typically discrete category labels. To meet the model input requirements, randomly initialized learnable vectors are used to represent each label, transforming it into a continuous and richer representation. Each label has a corresponding learnable vector, which is randomly initialized before model training and then updated using optimization methods such as gradient descent. The update of the label vector is driven by the gradient calculated by the backpropagation algorithm, which originates from the model's prediction error.

[0087] The second discrete information is obtained by concatenating the image code obtained from image modality coding and the tag code obtained from tag modality coding.

[0088] S400. Construct an initial Big Learning model based on the second discrete information. The initial Big Learning model represents an image classification and generation model. The second discrete information is masked by a random occlusion rate to construct the initial Big Learning model.

[0089] Specifically, refer to Figure 5 The initial Big Learning model is constructed using the acquired second discrete information. This Big Learning model is a unified learning framework developed in this invention for image classification and generation in scenarios with incomplete data. The Big Learning model enables accurate image classification and generation even in scenarios with missing training data. The initial Big Learning model is constructed by masking the second discrete information using a random occlusion rate, thus obtaining an image classification and generation model capable of handling various occlusion scenarios.

[0090] Optionally, constructing the initial Big Learning model based on the second discrete information specifically includes:

[0091] S410. Target data and source data are obtained based on the second discrete information and the random masking rate, wherein the target data represents masked data and the source data represents unmasked data;

[0092] S420, a network architecture for permutation language modeling based on BERT and XLNET;

[0093] A pre-trained objective function is obtained based on the permutation language modeling network architecture, the target data, and the source data;

[0094] S430. Determine the initial BigLearning model based on the permutation language modeling network architecture and the pre-training objective function.

[0095] Reference Figure 5-6 Specifically, BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language representation model. XLNET is a generalized autoregressive pre-language model. Source and target data are randomly selected: the masking rate is obtained through random sampling, enabling the construction of a comprehensive training task. The masked data is the target data, denoted as set . and The unmasked data is the source data, denoted as the set. Data set The sample length is N, and L represents the length index coordinate set L = (1, 2, ..., N). When image information is used as the source data and labels are used as the target data, it corresponds to an image classification task; when labels are used as the source data and image information is used as the target data, it corresponds to a conditional generation task; when the source data is an empty set and image information is used as the target data, it corresponds to an unconditional generation task. Even when the data is not complete (i.e., the entire set of image information and labels is available), image classification and image generation tasks can still be completed. This means that Big Learning models can naturally handle incomplete data situations, meaning that incomplete data information can be naturally applied in the corresponding tasks.

[0096] A network architecture for permutation language modeling was constructed: Referring to BERT and XLNET, a Big Learning-based network architecture for permutation language modeling was designed based on the Transformer / ViT (VisionTransformer) network. Specifically, drawing on the implementation of the BERT network architecture, input-output masks were used to satisfy the source data... and target data To meet diverse proportion requirements; by leveraging XLNET's two-stream attention mechanism, the model models the generation order of all target data during the forward computation process.

[0097] Constructing the loss function: Based on the ideas of Big Learning, the following pre-training objective function is proposed:

[0098]

[0099] in, and These are the source index subset and the target index subset, respectively (i.e., for a subset of for (a subset of) This is a subset of the source index after random sorting. A subset of the target index after random sorting; BigLearning By simultaneously analyzing the distribution of joint, conditional, and marginal data To perform modeling, that is Ω represents a set or A subset of the distribution, and the difference between the two distributions is measured based on the divergence KL. Represented as Indicates based on existing available indexes Information from the location, predicting the next Image encoding or supervisory labeling at the location; θ represents the corresponding parameterized model distribution, θ represents the model parameters, and X represents the data sample. This represents a data sample from a subset of the source index after random sorting. This represents a data sample of the target index subset after random sorting. The model is based on a categorical distribution. That is, it degenerates into the common cross-entropy loss function; E q(S′,T′) This represents the expectation given the true data distribution, where q(S′,T′) represents the true data distribution, and min... θ This represents finding the parameter θ that minimizes the objective function; A metric representing the distance between the actual data distribution and the distribution modeled by the model; Indicating comprehensive / random consideration and The difference between the two distributions is calculated under the given proportions.

[0100] Specifically, the core idea of ​​Big Learning is to utilize the information contained in the observation data in a detailed and comprehensive manner. Therefore, under the maximum likelihood learning paradigm: ① Based on a single shared model network p θ (X T′ |X S′ ), Comprehensive modeling X T′ The order of generation (i.e.) (All possible orders), comprehensively mining the information naturally contained in the data from the perspective of various generation orders; ② Unlike those with basic models that are often fixed and The proportion of Big Learning models the distribution of all (or many) joint, conditional, and marginal data, taking into account both comprehensive and stochastic approaches. and The proportion.

[0101] S500. The initial Big Learning model is trained based on the first discrete information to obtain the final Big Learning model.

[0102] Specifically, the initial Big Learning model is continuously trained using the first discrete information to obtain the final Big Learning model.

[0103] In one specific embodiment, the number of training epochs is 100, the batch size is 32, and the Adamw optimizer is used for training with parameters set to beta1 = 0.9, beta2 = 0.999, and epsilon = 1e-8.

[0104] The learning rate is initialized to 5e-5, and a warmup strategy is used in the initial stage of training, which involves linearly increasing the learning rate within a certain number of steps to avoid the model converging too quickly in the early stages of training.

[0105] Given a batch of image-label data pairs, the model is first constructed as input, then random masking is used to determine the source and target data of this batch, and then the network architecture is used to obtain the prediction result of the target data. Finally, the prediction error is calculated using the constructed loss function, and the model parameters are updated through backpropagation.

[0106] In one specific implementation, the Big Learning model is tested:

[0107] Reference Figure 7 Image classification task test: In the image classification task, image modal data is used as source data, and label modal data is used as target data. The classification accuracy is calculated using the test dataset.

[0108] Reference Figure 8 Image generation task testing: The generation tasks include conditional generation, unconditional generation, and missing data completion. In the conditional generation task, labeled modal data is used as the source data, and image modal data is used as the target data; in the unconditional generation task, only image modal data is used as the target data; in the missing data completion task, undisturbed image data is used as the source data, and missing image data is used as the target data.

[0109] In a specific embodiment, further verification was conducted through experiments:

[0110] S510, Image Classification Task: An image classification task experiment was conducted using SAR remote sensing images. Due to the complex electromagnetic imaging mechanism and the combined influence of numerous factors, SAR remote sensing images are not easily interpreted by humans. However, they contain rich and diverse semantic information, making SAR images a highly practical and challenging dataset for testing.

[0111] S511, Image-Label Bimodal Classification Experiment.

[0112] The dataset includes two categories: ships and aircraft. Ship data comes from the HRSID or RSDD ship datasets, while aircraft data comes from SADD on aircraft. All data observations are from the TerraSAR satellite. Image classification results are shown below. Figure 9 As shown, the model achieved a classification accuracy of 99.7% on the test set. As can be seen from the figure, SAR images are difficult to understand, but BigLearning's detailed multi-task construction and powerful data information mining capabilities help the model focus on and understand the essence of SAR images.

[0113] To further explore the internal mechanisms of the model, a heatmap of image classification for the BigLearning model was created from the perspective of model interpretability, such as... Figure 10 As shown in the figure, the model proposed in this invention can accurately capture the target, focusing on information related to the target's features.

[0114] To verify that Big Learning can be easily and directly extended to real-world scenarios with more modalities, radar elevation angle information from SAR images was introduced, expanding the aforementioned image-label two-modality model to a three-modality model. Image classification experiments were then conducted based on the MSTAR dataset. The dataset contains ten target categories and four elevation angle information types. The model achieved a classification accuracy of 95% on the test set. The classification results are shown below. Figure 11 As shown.

[0115] S520, Image Generation Task.

[0116] Based on the aforementioned SAR image dataset, we added generation tests using the MNIST dataset.

[0117] S521, Conditional / Unconditional Generation.

[0118] Figure 12 The results of the Big Learning model in generating unconditional images are shown. As can be seen from the figure, the generated results are highly similar to the original SAR images, further verifying the model's data understanding capabilities. Figure 13 This demonstrates the results of experiments conducted on the MNIST dataset, including the joint generation of p. θ (x), the result is as follows Figure 13 As shown in (a); conditional generation of p under given labels θ (x|y), the result is as follows Figure 13 As shown in (b).

[0119] Image completion experiments were conducted using the MNIST dataset and the CelebA high-resolution face dataset, respectively. The results are as follows: Figure 14 As shown. Figure 14 (a) and Figure 14 The experimental results in (b) show that the pre-trained base model has the ability to perform random missing data completion / generation and controlled missing data completion / generation, respectively.

[0120] Implementing the embodiments of the present invention has the following beneficial effects: A training method for an image classification and generation model in a data-incomplete scenario, as described in this embodiment, includes: acquiring the label and image information of the original image; inputting the image information into a VQ-VAE for processing to obtain first discrete information, wherein the VQ-VAE is used to convert the image information into encoded information; inputting the label and the image information into a data processing model to obtain second discrete information, wherein the data processing model is used to encode the label and the image information respectively to obtain label encoding and image encoding, wherein the second discrete information includes the label encoding and the image encoding; constructing an initial Big Learning model based on the second discrete information, wherein the initial Big Learning model represents the image classification and generation model, wherein the second discrete information is masked using a random occlusion rate to construct the initial Big Learning model; and training the initial Big Learning model based on the first discrete information to obtain a final Big Learning model. The first discrete information is obtained through VQ-VAE processing, and the second discrete information obtained through the data processing model is used to construct an initial Big Learning model. The initial Big Learning model is then trained using the first discrete information to obtain the final Big Learning model. This method can train a Big Learning model that accurately identifies objects and features in images even in scenarios with missing or incomplete data, thereby improving the accuracy of image classification and generation.

[0121] like Figure 15 As shown, this embodiment of the invention also provides a training system for image classification and generation models in scenarios with incomplete data, including:

[0122] The first module is used to obtain the labels and image information of the original image;

[0123] The second module is used to input the image information into VQ-VAE for processing to obtain the first discrete information, wherein the VQ-VAE is used to convert the image information into encoded information;

[0124] The third module is used to input the label and the image information into the data processing model to obtain the second discrete information. The data processing model is used to encode the label and the image information respectively to obtain the label code and the image code. The second discrete information includes the label code and the image code.

[0125] The fourth module is used to construct an initial Big Learning model based on the second discrete information. The initial Big Learning model represents an image classification and generation model. The second discrete information is used to construct the initial Big Learning model after being masked by a random masking rate.

[0126] The fifth module is used to train the initial Big Learning model based on the first discrete information to obtain the final Big Learning model.

[0127] It is evident that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0128] like Figure 16 As shown, this embodiment of the invention also provides a training apparatus for image classification and generation models in scenarios with incomplete data, comprising:

[0129] At least one processor;

[0130] At least one memory for storing at least one program;

[0131] When the at least one program is executed by the at least one processor, the at least one processor performs the method steps described in the above method embodiments.

[0132] It is evident that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented in this device embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0133] Furthermore, this application also discloses a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the described method. Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0134] It is understood that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0135] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A training method for image classification and generation models in scenarios with incomplete data, characterized in that, include: Obtain the label and image information of the original image; The image information is input into VQ-VAE for processing to obtain the first discrete information. The VQ-VAE is used to convert the image information into encoded information. After inputting the label and the image information into the data processing model, second discrete information is obtained. The data processing model is used to encode the label and the image information respectively to obtain label code and image code. The second discrete information includes the label code and the image code. An initial Big Learning model is constructed based on the second discrete information. The initial Big Learning model represents an image classification and generation model. The second discrete information is masked by a random masking rate to construct the initial Big Learning model. The step of constructing an initial Big Learning model based on the second discrete information specifically includes: obtaining target data and source data according to the second discrete information and the random masking rate, wherein the target data represents masked data and the source data represents unmasked data; constructing a permutation language modeling network architecture based on BERT and XLNET; obtaining a pre-trained objective function according to the permutation language modeling network architecture, the target data, and the source data; and determining the initial Big Learning model based on the permutation language modeling network architecture and the pre-trained objective function. The step of obtaining target data and source data based on the second discrete information and the random occlusion rate specifically includes: obtaining the random occlusion rate; and occluding the label encoding or the image encoding according to the random occlusion rate to obtain the target data and the source data. The initial Big Learning model is trained based on the first discrete information to obtain the final Big Learning model.

2. The method according to claim 1, characterized in that, After obtaining the target data and the source data by masking the label encoding or the image encoding according to the random masking rate, the method further includes: If the target data is the label encoding and the source data is the image encoding, then the image classification task is started; If the target data is the image encoding and the source data is the tag encoding, then the image condition generation task is started; If the target data is the image encoding and the source data is an empty set, then the unconditional image generation task is started.

3. The method according to claim 1, characterized in that, The permutation language modeling network architecture built based on BERT and XLNET specifically includes: A masking module is constructed based on the BERT, which is used to input and output masks to obtain different source data and target data. An arrangement module is constructed based on the XLNET, which is used to arrange the encoding of the target data; The permutation language modeling network architecture is obtained based on the permutation module and the mask module.

4. The method according to claim 1, characterized in that, The step of inputting the image information into VQ-VAE for processing to obtain the first discrete information specifically includes: The image information is input into the encoder of the VQ-VAE to obtain continuous encoded information; The first discrete information is obtained by mapping the continuous encoded information to the encoding cluster of the VQ-VAE.

5. The method according to claim 1, characterized in that, The step of inputting the label and the image information into the data processing model to obtain the second discrete information specifically includes: The image information is input into the data processing model for image modality encoding to obtain the image code; The label is input into the data processing model and subjected to label modal encoding to obtain the label code; The second discrete information is obtained by concatenating the image encoding and the label encoding.

6. A training system for image classification and generation models in scenarios with incomplete data, characterized in that, include: The first module is used to obtain the labels and image information of the original image; The second module is used to input the image information into VQ-VAE for processing to obtain the first discrete information, wherein the VQ-VAE is used to convert the image information into encoded information; The third module is used to input the label and the image information into the data processing model to obtain the second discrete information. The data processing model is used to encode the label and the image information respectively to obtain the label code and the image code. The second discrete information includes the label code and the image code. The fourth module is used to construct an initial Big Learning model based on the second discrete information. The initial Big Learning model represents an image classification and generation model. The second discrete information is used to construct the initial Big Learning model after being masked by a random masking rate. The step of constructing an initial Big Learning model based on the second discrete information specifically includes: obtaining target data and source data according to the second discrete information and the random masking rate, wherein the target data represents masked data and the source data represents unmasked data; constructing a permutation language modeling network architecture based on BERT and XLNET; obtaining a pre-trained objective function according to the permutation language modeling network architecture, the target data, and the source data; and determining the initial Big Learning model based on the permutation language modeling network architecture and the pre-trained objective function. The step of obtaining target data and source data based on the second discrete information and the random occlusion rate specifically includes: obtaining the random occlusion rate; and occluding the label encoding or the image encoding according to the random occlusion rate to obtain the target data and the source data. The fifth module is used to train the initial Big Learning model based on the first discrete information to obtain the final Big Learning model.

7. A training device for image classification and generation models in scenarios with incomplete data, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-5.

Images