A training method for an image classification model, an image classification method, and an apparatus.
By combining a pre-trained network for image reconstruction with text and image data training, the problem of obtaining high-quality labeled data is solved, thus improving the feature extraction and classification accuracy of the image classification model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LIANREN HEALTHCARE BIG DATA TECH CO LTD
- Filing Date
- 2023-07-14
- Publication Date
- 2026-05-26
AI Technical Summary
Existing medical imaging models rely on high-quality labeled data during training, which leads to high labeling costs and difficulty in obtaining available data, thus affecting model accuracy.
A pre-trained network for image-text reconstruction is adopted, which is combined with text reconstruction network and image reconstruction network. The network is trained using image-text data pairs, and the model is optimized using text loss function and image loss function. The encoding network of image reconstruction network is fused with the pre-set image classification model to form a new image classification model.
No labeled training data is required, which improves the feature extraction capability of the image classification model and increases the accuracy of image classification.
Smart Images
Figure CN116797884B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image analysis technology, and in particular to a training method for an image classification model, an image classification method, and an apparatus. Background Technology
[0002] In recent years, with the development of artificial intelligence, using medical image models for disease identification and classification has become one of the important directions for the development of intelligent healthcare.
[0003] Currently, supervised model training relies on a large amount of labeled data. However, high-quality medical labeled data is extremely difficult to obtain. It requires experienced imaging experts to annotate a large number of medical images with high quality. Moreover, the cost of image data annotation is high and the availability of data is difficult. The quality of labeled data directly affects the accuracy of the model. Summary of the Invention
[0004] This invention provides a training method, an image classification method, and an apparatus for an image classification model, in order to solve the problem of model accuracy being affected by annotation quality.
[0005] According to one aspect of the present invention, a method for training an image classification model is provided, comprising:
[0006] Obtain a pre-constructed image-text reconstruction pre-trained network; the image-text reconstruction pre-trained network includes a text reconstruction network and an image reconstruction network, the image reconstruction network includes a corresponding first encoding network branch and a first decoding network branch;
[0007] Obtain the training dataset of the image-text reconstruction pre-trained network; the training dataset is a training data pair formed by image data and the corresponding text data of the image data;
[0008] The image-text reconstruction pre-trained network is trained based on the training dataset to obtain the target image-text reconstruction network; the target loss function of the image-text reconstruction pre-trained network includes a text loss function and an image loss function.
[0009] The image classification model is determined based on the encoding network branches of the image reconstruction network in the target image reconstruction network and the preset image classification model.
[0010] According to one aspect of the present invention, an image classification method is provided, comprising:
[0011] Obtain the image to be classified;
[0012] The image to be classified is input into an image classification model for image classification to obtain the image classification result; the image classification model is trained according to the image classification model training method described in any embodiment of the present invention.
[0013] According to another aspect of the present invention, a training apparatus for an image classification model is provided, comprising:
[0014] The image and text reconstruction pre-trained network acquisition module is used to acquire a pre-constructed image and text reconstruction pre-trained network; the image and text reconstruction pre-trained network includes a text reconstruction network and an image reconstruction network, and the image reconstruction network includes a corresponding first encoding network branch and a first decoding network branch;
[0015] The training dataset acquisition module is used to acquire the training dataset of the image-text reconstruction pre-trained network; the training dataset is a training data pair formed by image data and the text data corresponding to the image data;
[0016] The target image-text reconstruction network training module is used to train the image-text reconstruction pre-trained network based on the training dataset to obtain the target image-text reconstruction network; the target loss function of the image-text reconstruction pre-trained network includes a text loss function and an image loss function;
[0017] The image classification model determination module is used to determine the image classification model based on the encoding network branch of the image reconstruction network in the target image reconstruction network and the preset image classification model.
[0018] According to another aspect of the present invention, an image classification apparatus is provided, comprising:
[0019] The image acquisition module is used to acquire images to be classified.
[0020] An image classification module is used to input the image to be classified into an image classification model for image classification and obtain an image classification result; the image classification model is trained according to the image classification model training method described in any embodiment of the present invention.
[0021] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0022] At least one processor; and
[0023] A memory communicatively connected to the at least one processor; wherein,
[0024] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method and / or image classification method of the image classification model according to any embodiment of the present invention.
[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the training method and / or image classification method of the image classification model according to any embodiment of the present invention.
[0026] The technical solution of this invention involves obtaining a pre-constructed image-text reconstruction pre-trained network. This network includes a text reconstruction network and an image reconstruction network, with the image reconstruction network comprising a first encoding network branch and a first decoding network branch. A training dataset for the pre-trained network is obtained, consisting of training data pairs formed by image data and corresponding text data. The pre-trained network is then trained on this dataset to obtain a target image-text reconstruction network. The target loss function of the pre-trained network includes a text loss function and an image loss function. An image classification model is determined based on the encoding network branch of the image reconstruction network in the target image-text reconstruction network and a preset image classification model. This eliminates the need for labeling the training data, solving the problem of model accuracy being affected by labeling quality. By fusing the encoding network in the target image reconstruction pre-trained network with the preset image classification model, a new image classification model is obtained, which can improve the feature extraction capability of the image classification model, thereby increasing the accuracy of the image classification model.
[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0029] Figure 1 This is a flowchart of a training method for an image classification model provided in Embodiment 1 of the present invention;
[0030] Figure 2 This is a schematic diagram of the structure of the image and text reconstruction pre-trained network provided in Embodiment 1 of the present invention;
[0031] Figure 3 This is a schematic diagram of image data before and after masking processing provided in Embodiment 1 of the present invention.
[0032] Figure 4 This is a flowchart of an image classification method provided in Embodiment 2 of the present invention;
[0033] Figure 5 This is a schematic diagram of the structure of a training device for an image classification model provided in Embodiment 3 of the present invention;
[0034] Figure 6 This is a schematic diagram of the structure of an image classification device provided in Embodiment 4 of the present invention;
[0035] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Detailed Implementation
[0036] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0038] Example 1
[0039] Figure 1 This is a flowchart of a training method for an image classification model provided in Embodiment 1 of the present invention. This embodiment is applicable to the training of image classification models. The method can be executed by an image classification model training device, which can be implemented in hardware and / or software. This image classification model training device can be configured in the electronic device provided in this embodiment of the present invention. Figure 1 As shown, the method includes:
[0040] S110. Obtain a pre-constructed image-text reconstruction pre-trained network; the image-text reconstruction pre-trained network includes a text reconstruction network and an image reconstruction network, and the image reconstruction network includes a corresponding first encoding network branch and a first decoding network branch.
[0041] Figure 2 This is a schematic diagram of the structure of the image-text reconstruction pre-trained network provided in Embodiment 1 of the present invention, as shown below. Figure 2 As shown, the image-text reconstruction pre-training network includes a text reconstruction network and an image reconstruction network. Further, the text reconstruction network includes a text encoding network and a text decoding network, and the image reconstruction network includes an image encoding network and an image decoding network, i.e., a first encoding branch and a first decoding branch. The text encoding network encodes the masked text data to obtain text encoding features; the image encoding network encodes the encoded image data to obtain image encoding features. The obtained text encoding features and image encoding features are fused to obtain fused text features. The text decoding network decodes the padded fused text features to obtain reconstructed text data; the image decoding network decodes the padded image encoding features to obtain reconstructed image data. It should be noted that feature fusion of text encoding features and image encoding features allows the image encoding features to be incorporated into the text encoding features, thereby enhancing the encoding capability of the trained image encoding network.
[0042] S120. Obtain the training dataset of the image-text reconstruction pre-trained network; the training dataset is a training data pair formed by image data and the text data corresponding to the image data.
[0043] The training dataset consists of image-text data pairs used to train the image-text reconstruction pre-trained network. Specifically, medical image data of any disease can be collected as image data, and text data corresponds to the image data. Medical image data includes, but is not limited to, ultrasound examination image data and MRI image data. For example, if the image data is an MRI image of breast cancer, then the text data is the image report of the MRI image of that breast cancer.
[0044] S130. The image-text reconstruction pre-trained network is trained based on the training dataset to obtain the target image-text reconstruction network; the target loss function of the image-text reconstruction pre-trained network includes a text loss function and an image loss function.
[0045] The target image-text reconstruction network refers to the pre-trained image-text reconstruction network. In this embodiment, the image-text reconstruction pre-trained network is trained using image-text data pairs as training data to obtain the target image-text reconstruction network. During the training of the image-text reconstruction pre-trained network, the target loss function used includes a text loss function and an image loss function. The text loss function is used to calculate the loss between the original text and the reconstructed text, and the image loss function is used to calculate the loss between the original image and the reconstructed image. The image loss function includes a first image reconstruction loss function and a second image reconstruction loss function.
[0046] For example, the formula for the target loss function is as follows:
[0047] Loss=αL MIM +(1-α)(L rec +L char )
[0048] Where Loss represents the target loss value, L MIM L represents text loss. rec For the reconstruction loss of the first image, L char α represents the loss for the second image reconstruction, and α is the weighting parameter.
[0049] Based on the above embodiments, optionally, the training process of the image and text reconstruction pre-trained network includes: iteratively executing the following training process until the image and text reconstruction pre-trained network reaches a preset training objective: acquiring training data pairs in the training sample set; performing masking processing on the training data pairs to obtain target training data pairs; training the image and text reconstruction pre-trained network based on the target training data pairs and the target loss function to obtain a target loss value; and adjusting the model parameters of the image and text reconstruction pre-trained network based on the target loss value.
[0050] Specifically, the training steps for the image-text reconstruction pre-trained network are as follows:
[0051] 1) Obtain image-text data pairs from the training sample set;
[0052] 2) Mask the image data and text data separately to obtain target training data pairs; where the target training data pairs are data pairs formed by masked image data and text data.
[0053] 3) Reconstruct the pre-trained network from the training images and text based on the target training data, and determine the target loss value based on the target loss function;
[0054] 4) Backpropagate the target loss value and adjust the model parameters of the image-text reconstruction pre-trained network;
[0055] 5) Iterate through steps 1-4 above until the image and text reconstruction pre-trained network reaches the preset training objective.
[0056] It is understandable that the preset training objective can be the convergence of the image reconstruction pre-trained network or the quality of the reconstructed image reaching a preset quality. The specific training objective can be set by those skilled in the art according to their needs, and is not limited here.
[0057] Based on the above embodiments, optionally, the target training data pair includes masked image patches and masked text; the masking process of the training data pair to obtain the target training data pair includes: segmenting the image data to obtain image patches of the image data; masking the image patches of the image data based on a preset ratio to obtain masked image patches; and masking the text data based on a preset ratio to obtain masked text.
[0058] In this embodiment, before encoding the image data and text data, masking processing needs to be performed on the image data and text data respectively. Specifically, the masking method for image data is to segment the image data into patches, and for each segmented patch, a portion of the patches is masked according to a preset ratio to obtain masked patches. Here, a masked patch refers to the patch after masking processing. For example, refer to... Figure 3 , Figure 3 This is a schematic diagram of image data before and after masking processing provided in Embodiment 1 of the present invention.
[0059] The text data masking method involves masking the text data according to a preset ratio to obtain masked text. For example, taking the image report of breast cancer MRI as text data, if the text data is "left breast mass, BI-RADS5, with enlarged left axillary lymph nodes", then the masked text can be "left breast [mask], BI-RADS5 [mask], with [mask] axillary [mask] enlargement".
[0060] It should be noted that the preset ratio for masking blocks and text can be the same or different, and the specific settings are determined by those skilled in the art; no restrictions are imposed here.
[0061] Based on the above embodiments, optionally, the text reconstruction network includes a second encoding network branch and a second decoding network branch; training the image-text reconstruction pre-training network based on the target training data pair and the target loss function to obtain a target loss value includes: inputting the masked image patch into the first encoding network branch for encoding to obtain image encoding features; and inputting the masked text into the second encoding network branch for encoding to obtain text encoding features; performing feature fusion based on the image encoding features and the text encoding features to obtain fused text encoding features; filling the mask positions of the fused text encoding features and inputting the filled fused text encoding features into the first decoding network branch for decoding to obtain reconstructed text data; and filling the image encoding features and inputting the filled image encoding features into the second decoding network branch for decoding to obtain reconstructed image data; determining the loss between the text data and the reconstructed text data based on the text loss function to obtain text loss; determining the loss between the image data and the reconstructed image data based on the image loss function to obtain image loss; and determining the target loss value based on the text loss and the image loss.
[0062] like Figure 2 As shown, during the training process of the image reconstruction pre-training network, the image loss is determined by inputting the masked image patch after masking into the image coding network for encoding processing to obtain image coding features; filling the image coding features and inputting the filled image coding features into the image decoding network for decoding processing to obtain reconstructed image data; and calculating the loss between the original image data and the reconstructed image data based on the image loss function to obtain the image loss.
[0063] The text loss is determined by inputting the masked text after masking into a text encoding network for encoding to obtain text encoding features; fusing the image encoding features and text encoding features to obtain fused text encoding features; inputting the fused text features into a text decoding network for decoding to obtain reconstructed text data; and calculating the loss between the original text data and the reconstructed text data based on the text loss function to obtain the text loss.
[0064] For example, the text loss function is as follows:
[0065]
[0066] Among them, L MLM Let N represent the text loss, N be the number of training samples, and T be the sequence length of the training samples. Let y be the predicted probability of the j-th position of the i-th training sample. i,jThis is used to indicate whether the j-th position of the i-th training sample is occluded. If it is occluded, then y i,j If y is 1, then if it is not obscured, then y i,j It is 0.
[0067] Based on the above embodiments, optionally, the image loss function includes a first image reconstruction loss function and a second image reconstruction loss function; the step of determining the loss between the image data and the reconstructed image data based on the image loss function to obtain the image loss includes: determining the loss between the image data and the reconstructed image data based on the first image reconstruction loss function to obtain a first image reconstruction loss; inputting the image data and the reconstructed image data into a pre-trained network model to obtain a first encoding feature and a second encoding feature; and determining the loss between the first encoding feature and the second encoding feature based on the second image reconstruction loss function to obtain a second image reconstruction loss.
[0068] The first image reconstruction loss is the L2 loss between the original image data and the reconstructed image data.
[0069] For example, the loss function for the first image reconstruction is as follows:
[0070]
[0071] Among them, L rec The loss is for the first image reconstruction, where N is the number of training samples, and x is the value of x. i Let x represent the i-th original image data. i ′ This represents the i-th reconstructed image data.
[0072] The second image reconstruction loss is the L1 loss between the first and second coding features. The first coding feature is the coding feature obtained by encoding the image data based on a pre-trained network model, and the second coding feature is the coding feature obtained by encoding the reconstructed image data based on a pre-trained network model.
[0073] For example, the pre-trained network model can be a three-dimensional residual neural network model, and the second reconstruction loss function is as follows:
[0074]
[0075] Among them, L char The loss is for the second image reconstruction, where N is the number of training samples, and 3DResNet(x) i ) represents the first encoded feature of the i-th original image data, 3DResNet(x) i ′) represents the second encoded feature of the i-th reconstructed image data.
[0076] S140. Determine the image classification model based on the encoding network branch of the image reconstruction network in the target image reconstruction network and the preset image classification model.
[0077] The preset image classification model is a pre-trained image classification model. In this embodiment, the preset image classification model is combined with the image encoding network of the image reconstruction network in the target image reconstruction network to obtain the image classification model. By combining the encoding network in the pre-trained target image reconstruction network with the preset image classification model, this embodiment obtains a new image classification model, which can improve the feature extraction capability of the image classification model, thereby improving the classification accuracy of the image classification model.
[0078] Based on the above embodiments, optionally, the step of determining the image classification model based on the encoding network branch and the preset classification network branch of the image reconstruction network in the target image reconstruction network includes: removing the text reconstruction network and the decoding network branch in the image reconstruction network from the target image reconstruction network to obtain the encoding network branch in the image reconstruction network; keeping the model parameters of the encoding network branch in the image reconstruction network unchanged; and combining the encoding network branch in the image reconstruction network with the preset image classification model to obtain the image classification model.
[0079] In this embodiment, the text reconstruction network in the target image reconstruction network and the image decoding network in the image reconstruction network are removed. That is, only the image encoding network in the image reconstruction network is retained, and the model parameters in the image encoding network remain unchanged. The image encoding network is combined with the preset image classification model, and the combined model is fine-tuned to obtain the image classification model.
[0080] The technical solution of this embodiment involves obtaining a pre-constructed image-text reconstruction pre-trained network. This network includes a text reconstruction network and an image reconstruction network, with the image reconstruction network comprising a first encoding network branch and a first decoding network branch. A training dataset for the image-text reconstruction pre-trained network is obtained. The training dataset consists of training data pairs formed by image data and corresponding text data. The image-text reconstruction pre-trained network is trained based on the training dataset to obtain a target image-text reconstruction network. The target loss function of the image-text reconstruction pre-trained network includes a text loss function and an image loss function. An image classification model is determined based on the encoding network branch of the image reconstruction network in the target image-text reconstruction network and a preset image classification model. This eliminates the need for labeling the training data, solving the problem of model accuracy being affected by labeling quality. By fusing the encoding network in the target image reconstruction pre-trained network with the preset image classification model, a new image classification model is obtained, which can improve the feature extraction capability of the image classification model, thereby increasing the accuracy of the image classification model.
[0081] Example 2
[0082] Figure 4 This is a flowchart of an image classification method provided in Embodiment 2 of the present invention. This embodiment is an image classification method based on the image classification model obtained in the above embodiments. Figure 4 As shown, the method includes:
[0083] S410. Obtain the image to be classified.
[0084] S420. Input the image to be classified into the image classification model to perform image classification and obtain the image classification result; the image classification model is trained by the image classification model training method according to any embodiment of the present invention.
[0085] The images to be classified include, but are not limited to, ultrasound images, MRI images, and other images requiring image classification. This is not a limitation; for example, the images to be classified could be MRI images of diseases such as cerebral hemorrhage, ischemic diseases, and brain tumors. In this embodiment, the image to be classified is used as model input and fed into the image classification model trained in the above embodiment for image classification to obtain the image classification result.
[0086] The technical solution of this embodiment involves acquiring an image to be classified; inputting the image to be classified into the training method of the image classification model according to any embodiment of the present invention to train the image classification model to obtain a prediction, thereby obtaining an image classification result; since the feature extraction capability of the image reconstruction model is improved when training the image classification model, the accuracy of the image classification result can be improved.
[0087] Example 3
[0088] Figure 5 This is a schematic diagram of the structure of a training device for an image classification model provided in Embodiment 3 of the present invention. Figure 5 As shown, the device includes:
[0089] The image and text reconstruction pre-trained network acquisition module 510 is used to acquire a pre-constructed image and text reconstruction pre-trained network; the image and text reconstruction pre-trained network includes a text reconstruction network and an image reconstruction network, and the image reconstruction network includes a corresponding first encoding network branch and a first decoding network branch;
[0090] The training dataset acquisition module 520 is used to acquire the training dataset of the image-text reconstruction pre-trained network; the training dataset is a training data pair formed by image data and the text data corresponding to the image data;
[0091] The target image-text reconstruction network training module 530 is used to train the image-text reconstruction pre-trained network based on the training dataset to obtain the target image-text reconstruction network; the target loss function of the image-text reconstruction pre-trained network includes a text loss function and an image loss function;
[0092] The image classification model determination module 540 is used to determine the image classification model based on the coding network branch of the image reconstruction network in the target image reconstruction network and the preset image classification model.
[0093] Based on the above embodiments, optionally, the target image-text reconstruction network training module 530 is used to train the image-text reconstruction pre-trained network. Specifically, the following training process is iteratively executed until the image-text reconstruction pre-trained network reaches the preset training target: acquiring training data pairs from the training sample set; masking the training data pairs to obtain target training data pairs; training the image-text reconstruction pre-trained network based on the target training data pairs and the target loss function to obtain a target loss value; and adjusting the model parameters of the image-text reconstruction pre-trained network based on the target loss value.
[0094] Based on the above embodiments, optionally, the target training data pair includes masked image patches and masked text; the target image-text reconstruction network training module 530 includes a masking unit, used to segment the image data to obtain image patches of the image data, perform masking processing on the image patches of the image data based on a preset ratio to obtain masked image patches; and perform masking processing on the text data based on a preset ratio to obtain masked text.
[0095] Based on the above embodiments, optionally, the text reconstruction network includes a second encoding network branch and a second decoding network branch; the target image-text reconstruction network training module 530 includes a target loss value determination unit, used to input the mask patch into the first encoding network branch for encoding to obtain image encoding features; and input the mask text into the second encoding network branch for encoding to obtain text encoding features; perform feature fusion based on the image encoding features and the text encoding features to obtain fused text encoding features; fill the mask positions of the fused text encoding features, and input the filled fused text encoding features into the first decoding network branch for decoding to obtain reconstructed text data; and fill the image encoding features, and input the filled image encoding features into the second decoding network branch for decoding to obtain reconstructed image data; determine the loss between the text data and the reconstructed text data based on the text loss function to obtain text loss; determine the loss between the image data and the reconstructed image data based on the image loss function to obtain image loss; and determine a target loss value based on the text loss and the image loss.
[0096] Based on the above embodiments, optionally, the image loss function includes a first image reconstruction loss function and a second image reconstruction loss function; the target loss value determination unit includes an image reconstruction loss determination subunit, used to determine the loss between the image data and the reconstructed image data based on the first image reconstruction loss function to obtain a first image reconstruction loss; input the image data and the reconstructed image data into a pre-trained network model to obtain a first encoding feature and a second encoding feature; determine the loss between the first encoding feature and the second encoding feature based on the second image reconstruction loss function to obtain a second image reconstruction loss.
[0097] Based on the above embodiments, optionally, the image classification model determination module 540 is specifically used to remove the text reconstruction network and the decoding network branch in the image reconstruction network of the target image reconstruction network to obtain the encoding network branch in the image reconstruction network; the model parameters of the encoding network branch in the image reconstruction network remain unchanged; and the encoding network branch in the image reconstruction network is combined with the preset image classification model to obtain the image classification model.
[0098] The image classification model training device provided in the embodiments of the present invention can execute the image classification model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0099] Example 4
[0100] Figure 6This is a schematic diagram of the structure of an image classification device provided in Embodiment 4 of the present invention. Figure 6 As shown, the device includes:
[0101] The image acquisition module 610 is used to acquire the image to be classified.
[0102] The image classification module 620 is used to input the image to be classified into the image classification model for image classification and obtain the image classification result; the image classification model is trained by the image classification model training method according to any embodiment of the present invention.
[0103] The image classification device provided in the embodiments of the present invention can execute the image classification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.
[0104] Example 5
[0105] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0106] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0107] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0108] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as training methods for image classification models and / or image classification methods.
[0109] In some embodiments, the training method and / or image classification method of the image classification model may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the training method and / or image classification method of the image classification model described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the training method and / or image classification method of the image classification model by any other suitable means (e.g., by means of firmware).
[0110] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0111] The computer program for training the image classification model and / or classifying the image classification method for implementing the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0112] Example 6
[0113] Embodiment 6 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a training method for an image classification model, the method comprising:
[0114] Obtain a pre-constructed image-text reconstruction pre-trained network; the image-text reconstruction pre-trained network includes a text reconstruction network and an image reconstruction network, the image reconstruction network includes a corresponding first encoding network branch and a first decoding network branch; obtain the training dataset of the image-text reconstruction pre-trained network; the training dataset consists of training data pairs formed by image data and corresponding text data; train the image-text reconstruction pre-trained network based on the training dataset to obtain the target image-text reconstruction network; the target loss function of the image-text reconstruction pre-trained network includes a text loss function and an image loss function; determine the image classification model based on the encoding network branch of the image reconstruction network in the target image-text reconstruction network and the preset image classification model.
[0115] And / or, perform an image classification method, which includes:
[0116] Obtain the image to be classified; input the image to be classified into the image classification model to perform image classification and obtain the image classification result; the image classification model is trained according to the image classification model training method of any embodiment of the present invention.
[0117] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0120] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0121] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A training method for an image classification model, characterized in that, include: Obtain a pre-constructed image-text reconstruction pre-trained network; the image-text reconstruction pre-trained network includes a text reconstruction network and an image reconstruction network, the image reconstruction network includes a corresponding first encoding network branch and a first decoding network branch; Obtain the training dataset of the image-text reconstruction pre-trained network; the training dataset is a training data pair formed by image data and the corresponding text data of the image data; Based on the training dataset, target training data pairs are determined. The image-text reconstruction pre-training network is trained based on the target training data pairs and the target loss function to obtain the target loss value. The model parameters of the image-text reconstruction pre-training network are adjusted based on the target loss value to obtain the target image-text reconstruction network. The target training data pairs include masked image patches and masked text. The target loss function of the image-text reconstruction pre-training network includes a text loss function and an image loss function. Remove the text reconstruction network and the decoding network branches from the target image reconstruction network to obtain the encoding network branch in the image reconstruction network; the model parameters of the encoding network branch in the image reconstruction network remain unchanged. The image classification model is obtained by combining the coding network branch in the image reconstruction network with the preset image classification model. The text reconstruction network includes a second encoding network branch and a second decoding network branch; the step of training the image-text reconstruction pre-trained network based on the target training data pair and the target loss function to obtain the target loss value includes: The masked image is input into the first encoding network branch for encoding to obtain image encoding features; and the masked text is input into the second encoding network branch for encoding to obtain text encoding features. Based on the image coding features and the text coding features, feature fusion is performed to obtain fused text coding features; The mask positions of the fused text encoding features are filled, and the filled fused text encoding features are input to the first decoding network branch for decoding to obtain reconstructed text data; and the image encoding features are filled, and the filled image encoding features are input to the second decoding network branch for decoding to obtain reconstructed image data. Based on the text loss function, the loss between the text data and the reconstructed text data is determined to obtain the text loss; Based on the image loss function, the loss between the image data and the reconstructed image data is determined to obtain the image loss; The target loss value is determined based on the text loss and the image loss.
2. The method according to claim 1, characterized in that, The training process of the image and text reconstruction pre-trained network includes: The following training process is executed iteratively until the image-text reconstruction pre-trained network reaches the preset training objective: Obtain training data pairs from the training sample set; The training data pairs are masked to obtain the target training data pairs; The image and text reconstruction pre-training network is trained based on the target training data pair and the target loss function to obtain the target loss value; Adjust the model parameters of the image reconstruction pre-trained network based on the target loss value.
3. The method according to claim 2, characterized in that, The process of masking the training data pairs to obtain the target training data pairs includes: The image data is segmented to obtain image data patches, and the image data patches are masked based on a preset ratio to obtain masked image patches; The text data is masked based on a preset ratio to obtain masked text.
4. The method according to claim 1, characterized in that, The image loss function includes a first image reconstruction loss function and a second image reconstruction loss function; the step of determining the loss between the image data and the reconstructed image data based on the image loss function to obtain the image loss includes: Based on the first image reconstruction loss function, the loss between the image data and the reconstructed image data is determined to obtain the first image reconstruction loss; The image data and the reconstructed image data are input into a pre-trained network model to obtain a first encoding feature and a second encoding feature; the loss between the first encoding feature and the second encoding feature is determined based on the second image reconstruction loss function to obtain the second image reconstruction loss.
5. An image classification method, characterized in that, include: Obtain the image to be classified; The image to be classified is input into an image classification model to perform image classification and obtain the image classification result. The image classification model is trained using the image classification model training method according to any one of claims 1-4.
6. A training device for an image classification model, characterized in that, include: The image and text reconstruction pre-trained network acquisition module is used to acquire a pre-constructed image and text reconstruction pre-trained network; the image and text reconstruction pre-trained network includes a text reconstruction network and an image reconstruction network, and the image reconstruction network includes a corresponding first encoding network branch and a first decoding network branch; The training dataset acquisition module is used to acquire the training dataset of the image-text reconstruction pre-trained network; the training dataset is a training data pair formed by image data and the text data corresponding to the image data; The target image-text reconstruction network training module is used to determine target training data pairs based on the training dataset, train the image-text reconstruction pre-trained network based on the target training data pairs and the target loss function to obtain the target loss value, and adjust the model parameters of the image-text reconstruction pre-trained network based on the target loss value to obtain the target image-text reconstruction network; the target training data pairs include masked image patches and masked text, and the target loss function of the image-text reconstruction pre-trained network includes a text loss function and an image loss function; The image classification model determination module is used to remove the text reconstruction network and the decoding network branch in the image reconstruction network from the target image reconstruction network to obtain the encoding network branch in the image reconstruction network; the model parameters of the encoding network branch in the image reconstruction network remain unchanged; and the encoding network branch in the image reconstruction network is combined with a preset image classification model to obtain an image classification model. The text reconstruction network includes a second encoding network branch and a second decoding network branch. The target image-text reconstruction network training module includes a target loss value determination unit, which is used to input the masked image patch into the first encoding network branch for encoding to obtain image encoding features; input the masked text into the second encoding network branch for encoding to obtain text encoding features; perform feature fusion based on the image encoding features and the text encoding features to obtain fused text encoding features; fill the mask positions of the fused text encoding features and input the filled fused text encoding features into the first decoding network branch for decoding to obtain reconstructed text data; fill the image encoding features and input the filled image encoding features into the second decoding network branch for decoding to obtain reconstructed image data; determine the loss between the text data and the reconstructed text data based on the text loss function to obtain text loss; determine the loss between the image data and the reconstructed image data based on the image loss function to obtain image loss; and determine the target loss value based on the text loss and the image loss.
7. An image classification device, characterized in that, include: The image acquisition module is used to acquire images to be classified. An image classification module is used to input the image to be classified into an image classification model for image classification and obtain an image classification result; the image classification model is trained according to the training method of the image classification model according to any one of claims 1-4.
8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the image classification model according to any one of claims 1-4 and / or the image classification method according to claim 5.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the training method of the image classification model according to any one of claims 1-4 and / or the image classification method according to claim 5.