Training methods, devices and equipment for image classification models
By utilizing a neural network model to determine the category prediction results and calculate the loss during image classification model training, the problem of poor classification performance of image classification models is solved, and higher classification accuracy is achieved.
Patent Information
- Application Number
- CN202211210860.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-30
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-09-30
AI Technical Summary
The image classification models trained in the current technology do not perform well, which affects the accuracy of the image classification results.
By acquiring modality and category annotation results of multiple sample source and target domain images, a neural network model is used to determine the category prediction result. The loss is calculated based on the image features corresponding to the category annotation results, and the neural network model is trained to bring the image features of different modality annotation results with the same category annotation results closer together and weaken the modality difference.
It improves the accuracy of image classification model results by aligning image features in the source and target domains, making image features more conducive to distinguishing categories and weakening modal differences, thereby improving classification performance.
Smart Images

Figure CN117011571B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a training method, apparatus, and device for an image classification model. Background Technology
[0002] With the continuous development of computer technology, image classification technology has been widely applied in various industries. By training an image classification model, images can be classified.
[0003] In related technologies, when training an image classification model, the category labels of the sample source domain image, the sample target domain image, and the sample source domain image can be obtained. The sample source domain image and the sample target domain image are input into a neural network model, which determines the category prediction results for the sample source domain image and the sample target domain image. Based on the category prediction results and category labels of the sample source domain image, and the category prediction results of the sample target domain image, the neural network model is trained to obtain the image classification model.
[0004] However, the image classification model trained in the above manner does not perform well, which affects the accuracy of the image classification results. Summary of the Invention
[0005] This application provides a training method, apparatus, and device for an image classification model, which can be used to solve problems in related technologies. The technical solution includes the following contents.
[0006] On the one hand, a method for training an image classification model is provided, the method comprising:
[0007] Obtain multiple source domain images, multiple target domain images, modality annotation results for each source domain image, and category annotation results for each source domain image;
[0008] The first category prediction result of each sample source domain image is determined by a neural network model;
[0009] The neural network model is used to determine the first category prediction result for each sample target domain image;
[0010] For any category labeling result, based on each source domain image feature pair corresponding to the any category labeling result, a first loss corresponding to the any category labeling result is determined. The source domain image feature pair corresponding to the any category labeling result includes the image features of the sample source domain images of the two different modal labeling results corresponding to the any category labeling result.
[0011] Based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, the neural network model is trained to obtain an image classification model.
[0012] On the other hand, a training apparatus for an image classification model is provided, the apparatus comprising:
[0013] The acquisition module is used to acquire multiple sample source domain images, multiple sample target domain images, modality annotation results of each sample source domain image, and category annotation results of each sample source domain image;
[0014] A determination module is used to determine the first category prediction result of each sample source domain image through a neural network model;
[0015] The determining module is further configured to determine the first category prediction result of each sample target domain image through the neural network model;
[0016] The determining module is further configured to, for any category labeling result, determine a first loss corresponding to the any category labeling result based on each source domain image feature pair corresponding to the any category labeling result, wherein a source domain image feature pair corresponding to the any category labeling result includes image features of sample source domain images of two different modal labeling results corresponding to the any category labeling result;
[0017] The training module is used to train the neural network model based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, to obtain an image classification model.
[0018] In one possible implementation, the determining module is configured to, for any given source domain image, generate a first target domain image corresponding to the given source domain image based on the image features of the given source domain image using the neural network model; and determine a first category prediction result for the given source domain image based on the first target domain image corresponding to the given source domain image using the neural network model.
[0019] In one possible implementation, the determining module is configured to generate target domain features and image content features corresponding to the arbitrary source domain image based on the image features of the arbitrary source domain image; and generate a first target domain image corresponding to the arbitrary source domain image based on the target domain features and image content features corresponding to the arbitrary source domain image.
[0020] In one possible implementation, the determining module is configured to, for any sample target domain image, generate a first source domain image corresponding to the any sample target domain image based on the image features of the any sample target domain image using the neural network model; and determine a first category prediction result for the any sample target domain image based on the first source domain image corresponding to the any sample target domain image using the neural network model.
[0021] In one possible implementation, the training module is used to determine a second loss based on the first category prediction result of each sample source domain image and the category labeling result of each sample source domain image; determine a third loss based on the first category prediction result of each sample target domain image; and train the neural network model based on the second loss, the third loss, and the first loss corresponding to each category labeling result to obtain an image classification model.
[0022] In one possible implementation, the training module is configured to determine the weight coefficients of each sample target domain image based on the first category prediction result of each sample target domain image; determine the weight coefficients of each sample source domain image based on the weight coefficients of each sample target domain image; and determine a third loss based on the weight coefficients of each sample target domain image and the weight coefficients of each sample source domain image.
[0023] In one possible implementation, the first category prediction result of any sample target domain image includes the first probability that the sample target domain image belongs to a category other than the respective category annotation results;
[0024] The training module is used to obtain the second category prediction result of each sample target domain image, wherein the second category prediction result of any sample target domain image includes the second probability that the sample target domain image belongs to each category labeling result; for any sample target domain image, the weight coefficient of the sample target domain image is determined based on the first probability and the second probability corresponding to the sample target domain image.
[0025] In one possible implementation, the training module is configured to determine multiple reference images from the target domain images based on the weight coefficients of the target domain images, wherein the weight coefficients of the reference images are greater than a weight threshold; and for any source domain image, to determine the weight coefficients of the source domain image based on a second probability that the reference images belong to the category labeling results of the source domain image.
[0026] In one possible implementation, the training module is configured to acquire the domain prediction results of each sample target domain image and the domain prediction results of the first target domain image corresponding to each sample source domain image; and determine a third loss based on the weight coefficients of each sample target domain image, the domain prediction results of each sample target domain image, the weight coefficients of each sample source domain image, and the domain prediction results of the first target domain image corresponding to each sample source domain image.
[0027] In one possible implementation, the device further includes:
[0028] The determining module is further configured to determine the modality prediction result of each sample source domain image through the neural network model; and determine the fourth loss based on the modality prediction result and modality annotation result of each sample source domain image.
[0029] The training module is used to train the neural network model based on the fourth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, to obtain an image classification model.
[0030] In one possible implementation, the device further includes:
[0031] The determining module is further configured to obtain the domain prediction result of the first target domain image corresponding to each sample source domain image and the domain prediction result of each sample target domain image; and determine the fifth loss based on the domain prediction result of the first target domain image corresponding to each sample source domain image and the domain prediction result of each sample target domain image.
[0032] The training module is used to train the neural network model based on the fifth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, to obtain an image classification model.
[0033] In one possible implementation, the device further includes:
[0034] The determining module is further configured to generate a second source domain image corresponding to each sample source domain image based on the first target domain image corresponding to each sample source domain image; and determine a sixth loss based on each sample source domain image and the second source domain image corresponding to each sample source domain image.
[0035] The training module is used to train the neural network model based on the sixth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, to obtain an image classification model.
[0036] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the training method of any of the image classification models described above.
[0037] On the other hand, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to enable an electronic device to implement the training method of any of the above-described image classification models.
[0038] On the other hand, a computer program or computer program product is also provided, wherein at least one computer program is stored in the computer program or computer program product, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement the training method of any of the above-mentioned image classification models.
[0039] The technical solution provided in this application brings at least the following beneficial effects:
[0040] The technical solution provided in this application involves using the image features of the source domain images of two different modalities corresponding to a given category labeling result as a source domain image feature pair. This facilitates the comparison of image features of the source domain images of two sample source domain images with the same category labeling result but different modalities. A first loss corresponding to the category labeling result is determined using each source domain image feature pair, and an image classification model is trained using this first loss. This allows the image classification model to narrow down the image features of sample source domain images with the same category labeling result but different modalities, resulting in similar image features for sample source domain images with the same category labeling result but different modalities. This makes the image features more focused on distinguishing categories and less focused on distinguishing modalities. Therefore, the image features extracted by the image classification model are more conducive to image classification, improving the accuracy of the classification results. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a schematic diagram of a training method for an image classification model or an implementation environment for an image classification method provided in an embodiment of this application;
[0043] Figure 2 This is a flowchart of a training method for an image classification model provided in an embodiment of this application;
[0044] Figure 3 This is a schematic diagram illustrating the category relationship between an image in the source domain and an image in the target domain, provided in an embodiment of this application.
[0045] Figure 4 This is a flowchart of an image classification method provided in an embodiment of this application;
[0046] Figure 5 This is a schematic diagram illustrating the training process of an image classification model provided in an embodiment of this application;
[0047] Figure 6 This is a schematic diagram of an image classification method provided in an embodiment of this application;
[0048] Figure 7 This is a schematic diagram of the structure of a training device for an image classification model provided in an embodiment of this application;
[0049] Figure 8 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application;
[0050] Figure 9 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;
[0051] Figure 10 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation
[0052] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0053] Figure 1 This is a schematic diagram illustrating a training method for an image classification model or an implementation environment for an image classification method provided in this application embodiment, such as... Figure 1As shown, the implementation environment includes a terminal device 101 and a server 102. The training method or image classification method of the image classification model in this embodiment can be executed by the terminal device 101, by the server 102, or jointly by both the terminal device 101 and the server 102.
[0054] Terminal device 101 can be a smartphone, game console, desktop computer, tablet computer, laptop computer, smart TV, smart in-vehicle device, smart voice interaction device, smart home appliance, etc. Server 102 can be a single server, a server cluster consisting of multiple servers, or any of the following: cloud computing platform and virtualization center. This application embodiment does not limit this. Server 102 can communicate with terminal device 101 via a wired network or wireless network. Server 102 can have functions such as data processing, data storage, and data transmission and reception. This application embodiment does not limit this. The number of terminal devices 101 and servers 102 is not limited and can be one or more.
[0055] The image classification model training method or image classification method provided in this application embodiment can be implemented based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions.
[0056] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.
[0057] Image classification technology is a very important technology that can be applied to various industries. Typically, trained image classification models are used to classify images.
[0058] In related technologies, source domain images, target domain images, and category annotation results of the source domain images can be obtained. The category prediction results of the source domain image and the target domain image are determined using a neural network model. Based on the category prediction results and category annotation results of the source domain image, and the category prediction results of the target domain image, the neural network model is trained to obtain an image classification model. However, the classification performance of the trained image classification model is poor, which affects the accuracy of the image classification results.
[0059] It should be noted that "source domain" and "target domain" are terms from the field of transfer learning. Transfer learning refers to using existing knowledge to learn new knowledge, with the aim of finding similarities between existing and new knowledge. In transfer learning, the existing knowledge belongs to the source domain, and the new knowledge to be learned belongs to the target domain. The source and target domains are different but related. The goal is to reduce the distributional differences between the source and target domains to achieve knowledge transfer and ultimately data labeling. The image classification model training method provided in this application falls under the category of transfer learning.
[0060] This application provides a method for training an image classification model. This method can be applied to the above-described implementation environment, and the image classification model trained using this method can improve classification accuracy. Figure 2 The flowchart shown in this embodiment of the application illustrates a training method for an image classification model. This method can be implemented by... Figure 1 The method can be executed by either terminal device 101 or server 102, or jointly by both. For ease of description, the terminal device 101 or server 102 executing the training method of the image classification model in this embodiment is referred to as an electronic device, and this method can be executed by an electronic device. Figure 2 As shown, the method includes the following steps.
[0061] Step 201: Obtain multiple sample source domain images, multiple sample target domain images, modality annotation results of each sample source domain image, and category annotation results of each sample source domain image.
[0062] The sample source domain image is an image from the source domain, and the sample target domain image is an image from the target domain. The source domain and target domain are two different domains. This application does not limit the source domain and target domain; for example, the source domain is the painting domain, and the target domain is the real-world domain. Multiple sample source domain images and multiple sample target domain images can be obtained through manual input, web scraping, or other methods.
[0063] Each sample source domain image corresponds to a modality. The modalities of any two sample source domain images can be the same or different. In this embodiment, a modality can characterize the image style of an image in a certain domain. This embodiment does not limit the modality; for example, if the source domain is painting, the modality of the sample source domain images includes any modality such as oil painting, watercolor painting, or sketching. Modality annotation can be performed on any sample source domain image to obtain the modality annotation result. The modality annotation result of the sample source domain image can be understood as the modality label of the sample source domain image, used to reflect the modality of the image style to which the sample source domain image belongs. This embodiment does not limit the method of modality annotation; for example, modality annotation of the sample source domain images can be performed through a modality classification model or manual annotation.
[0064] Each sample source domain image corresponds to a category. Any two sample source domain images can belong to the same or completely different categories. In this embodiment, the category reflects the image content. This embodiment does not limit the category; for example, the categories of sample source domain images include animals, buildings, vehicles, etc. Category labeling can be performed on any sample source domain image to obtain the category labeling result. The category labeling result of the sample source domain image can be understood as the category label of the sample source domain image, used to reflect the category of the image content to which the sample source domain image belongs. This embodiment does not limit the method of category labeling; for example, category labeling of sample source domain images can be performed through image classification models or manual labeling.
[0065] Step 202: Determine the first category prediction result for each sample source domain image using a neural network model.
[0066] Any sample source domain image can be input into a neural network model. The neural network model then classifies the sample source domain image to obtain a first category prediction result. Since each sample source domain image corresponds to one category labeling result, and the category labeling results of any two sample source domain images can be the same or different, there are multiple category labeling results. In this embodiment, the first category prediction result of the sample source domain image includes the probability that the sample source domain image belongs to each category labeling result.
[0067] This application does not limit the model structure, size, etc. of the neural network model. For example, the neural network model includes a feature extractor and a first category classifier. The feature extractor is used to extract features from the sample source domain image, and the first category classifier is used to classify the sample source domain image.
[0068] Optionally, any sample source domain image is input into a neural network model, and the feature extractor in the neural network model extracts features from the sample source domain image to obtain the image features of the sample source domain image. For example, the feature extractor includes at least one concatenated convolutional block, where the sample source domain image can be regarded as the input feature of the first convolutional block, and the image features of the sample source domain image can be regarded as the output feature of the last convolutional block.
[0069] Any convolutional block includes a Convolutional Neural Network (CNN). The CNN performs convolution processing on the input features of the convolutional block to reduce the feature dimension of the input features. After the CNN, network layers such as pooling layers and activation layers can be concatenated. The pooling layer is used to pool the input features of the pooling layer, and the activation layer is used to activate the input features of the activation layer.
[0070] In one possible implementation, step 202 includes steps 2021 to 2022.
[0071] Step 2021: For any sample source domain image, generate the first target domain image corresponding to the sample source domain image based on the image features of the sample source domain image using a neural network model.
[0072] In this embodiment, the neural network model includes a first generator. The first generator is concatenated after the feature extractor. The first generator is a generator with a built-in attention mechanism, which can learn the mapping relationship between images in the source domain and images in the target domain. After the feature extractor extracts the image features of any sample source domain image, the generator maps the sample source domain image to the corresponding first target domain image based on the image features of the sample source domain image.
[0073] The sample source domain image is an image of the source domain, while the first target domain image corresponding to the sample source domain image is an image of the target domain. Therefore, the first target domain image corresponding to the sample source domain image belongs to a different domain than the sample source domain image.
[0074] Optionally, step 2021 includes: generating target domain features and image content features corresponding to any sample source domain image based on the image features of any sample source domain image; and generating a first target domain image corresponding to any sample source domain image based on the target domain features and image content features corresponding to any sample source domain image.
[0075] The first generator can decompose the image features of any sample source domain image into source domain features and image content features. Source domain features characterize the common characteristics of images within the source domain. For example, if the source domain is painting, images in painting often lack realistic textures and have strong lines; therefore, the source domain features reflect the lines and textures of the source domain images. Image content features characterize the image content of the sample source domain image, focusing more on describing the objects within it. For example, if the sample source domain image is about a face, its image content features will reflect facial information such as the eyes, nose, mouth, and eyebrows, focusing more on describing the face.
[0076] Since the first generator learns the mapping relationship between the source domain image and the target domain image, it can map the source domain features of the sample source domain image to the corresponding target domain features based on this mapping relationship. The target domain features corresponding to the sample source domain image are used to characterize the common features of the target domain image. For example, if the target domain is the real world, images in the real world mostly have realistic textures and weak lines. Therefore, the target domain features corresponding to the sample source domain image can reflect the lines and textures of the target domain image.
[0077] Next, the first generator can fuse the target domain features corresponding to any sample source domain image with the image content features of the sample source domain image, thereby fusing the common features of the image content of the sample source domain image and the image of the target domain to obtain the image features of the first target domain image corresponding to the sample source domain image, thereby generating the first target domain image corresponding to the sample source domain image.
[0078] For example, the process of generating the first target domain image corresponding to the sample source domain image can be represented as follows: Where x represents the source domain image of the sample, A represents the target domain feature corresponding to the source domain image of the sample, and C represents the image content feature of the source domain image of the sample. The first target domain image that represents the source domain image of the sample. The first generator is characterized by its attention mechanism, which can transform an image from the source domain into an image from the target domain.
[0079] Optionally, the first generator multiplies the target domain features corresponding to the sample source domain image with the image content features of the sample source domain image to obtain a first product result, and multiplies the image features of the sample source domain image with the target domain features corresponding to the sample source domain image to obtain a second product result. The sum of the first product result and the image features of the sample source domain image is subtracted from the information of the second product result to obtain the image features of the first target domain image corresponding to the sample source domain image, thereby determining the first target domain image corresponding to the sample source domain image. In this way, the first target domain image corresponding to the sample source domain image can be made closer to the sample source domain image in terms of image content, thus controlling the difference between the first target domain image corresponding to the sample source image and the sample source image as much as possible at the domain level.
[0080] For example, the first target domain image corresponding to the sample source domain image can be represented as: in, The first target domain image, C, represents the source domain image of the sample. s→t A represents the image content features of the sample source domain image. s→t The target domain features corresponding to the sample source domain image are represented by x, where x represents the sample source domain image. It is understood that the first generator can extract image features from the sample source domain image, and can also determine the first target domain image corresponding to the sample source domain image based on the image features of the first target domain image corresponding to the sample source domain image. Therefore, the process of extracting image features and the process of generating images based on image features are omitted here.
[0081] Step 2022: Determine the first category prediction result of any sample source domain image based on the first target domain image corresponding to any sample source domain image using a neural network model.
[0082] In this embodiment, a first-category classifier is concatenated after a first generator. The first target domain image corresponding to any sample source domain image generated by the first generator, or the image features of that first target domain image, is used as input information for the first-category classifier. The first-category classifier classifies the first target domain image based on the input information to obtain the first-category prediction result for the sample source domain image. This embodiment does not limit the model structure or size of the first-category classifier.
[0083] Step 203: Determine the first category prediction result of the target domain image of each sample through a neural network model.
[0084] Any sample target domain image can be input into a neural network model. The neural network model will classify the sample target domain image to obtain the first category prediction result of the sample target domain image. The first category prediction result of the sample target domain image includes the probability that the sample target domain image belongs to each category label result.
[0085] Optionally, any sample target domain image is input into a neural network model, and the feature extractor in the neural network model extracts features from the sample target domain image to obtain the image features of the sample target domain image. For example, the feature extractor includes at least one concatenated convolutional block, where the sample target domain image can be regarded as the input feature of the first convolutional block, and the image features of the sample target domain image can be regarded as the output feature of the last convolutional block.
[0086] In one possible implementation, step 203 includes steps 2031 to 2032.
[0087] Step 2031: For any sample target domain image, generate the first source domain image corresponding to the sample target domain image based on the image features of the sample target domain image using a neural network model.
[0088] In this embodiment of the application, the first generator learns the mapping relationship between the source domain image and the target domain image. After the feature extractor extracts the image features of any sample target domain image, the sample target domain image is mapped to the first source domain image corresponding to the sample target domain image based on the image features of the sample target domain image.
[0089] The sample target domain image is an image of the target domain, while the corresponding first source domain image is an image of the source domain. Therefore, the first source domain image corresponding to the sample target domain image belongs to a different domain than the sample target domain image.
[0090] Optionally, step 2031 includes: generating source domain features and image content features corresponding to any sample target domain image based on the image features of any sample target domain image; and generating a first source domain image corresponding to any sample target domain image based on the source domain features and image content features corresponding to any sample target domain image.
[0091] The first generator can decompose the image features of any sample target domain image into target domain features and image content features. The target domain features are used to characterize the common characteristics of the target domain image, while the image content features are used to characterize the image content, focusing more on describing the objects in the sample target domain image.
[0092] Since the first generator learns the mapping relationship between the source domain image and the target domain image, it can map the target domain features of the sample target domain image to the corresponding source domain features based on this mapping relationship. The source domain features corresponding to the sample target domain image are used to characterize the common features of the source domain images.
[0093] Next, the first generator can fuse the source domain features corresponding to any sample target domain image with the image content features of the sample target domain image, thereby fusing the common features of the image content of the sample target domain image and the image of the source domain to obtain the image features of the first source domain image corresponding to the sample target domain image, thereby generating the first source domain image corresponding to the sample target domain image.
[0094] For example, the process of generating the first source domain image corresponding to the sample target domain image can be represented as follows: Where x represents the target domain image of the sample, A represents the source domain feature corresponding to the target domain image of the sample, and C represents the image content feature of the target domain image of the sample. The first source domain image that represents the target domain image of the sample. The first generator is characterized by its attention mechanism, which can transform an image in the target domain into an image in the source domain.
[0095] Optionally, the first generator multiplies the source domain features corresponding to the sample target domain image with the image content features of the sample target domain image to obtain a third product result, and multiplies the image features of the sample target domain image with the source domain features corresponding to the sample target domain image to obtain a fourth product result. The sum of the third product result and the image features of the sample target domain image is then subtracted from the information of the fourth product result to obtain the image features of the first source domain image corresponding to the sample target domain image, thereby determining the first source domain image corresponding to the sample target domain image. In this way, the first source domain image corresponding to the sample target domain image can be made closer to the sample target domain image in terms of image content, thus controlling the difference between the first source domain image corresponding to the sample target domain image and the sample target domain image as much as possible at the domain level.
[0096] For example, the first source domain image corresponding to the sample target domain image can be represented as: in, The first source domain image, C, represents the target domain image of the sample. t→s A represents the image content features of the target domain image of the sample. t→sThe source domain features corresponding to the sample target domain image are represented by x, and the sample target domain image is represented by x. It can be understood that the first generator can extract the image features of the sample target domain image, or it can determine the first source domain image corresponding to the sample target domain image based on the image features of the first source domain image corresponding to the sample target domain image. Therefore, the process of extracting image features and the process of generating images based on image features are omitted here.
[0097] Step 2032: Determine the first category prediction result of any sample target domain image based on the first source domain image corresponding to any sample target domain image using a neural network model.
[0098] In this embodiment, the first source domain image corresponding to any sample target domain image generated by the first generator, or the image features of the first source domain image, is used as the input information of the first category classifier. The first category classifier classifies the first source domain image based on the input information to obtain the first category prediction result of the sample target domain image.
[0099] Step 204: For any category labeling result, based on each source domain image feature pair corresponding to the category labeling result, determine the first loss corresponding to the category labeling result. A source domain image feature pair corresponding to any category labeling result includes the image features of the sample source domain images of two different modal labeling results corresponding to the category labeling result.
[0100] In this embodiment of the application, any category labeling result corresponds to multiple sample source domain images. For any two sample source domain images corresponding to the category labeling result, if the modal labeling results of the two sample source domain images are different, then the image features of the two sample source domain images are used as a source domain image feature pair corresponding to the category labeling result.
[0101] For any source domain image feature pair corresponding to a category labeling result, calculate the feature distance between the image features of the two sample source domain images in the source domain image feature pair. This feature distance can be cosine distance, Euclidean distance, least square error (also known as L2 norm), the square of L2 norm, etc. This feature distance can be simply referred to as the distance of the source domain image feature pair.
[0102] For example, the source domain images of each sample corresponding to the i-th category labeling result can be characterized as Here, m represents the number of modality annotation results corresponding to the i-th category annotation result. In this case, the sample source domain image The category labeling result is the i-th category labeling result, and the modality labeling result is the 1-th modality labeling result; sample source domain image The category labeling result is the i-th category labeling result, and the modality labeling result is the m-th modality labeling result.
[0103] Based on the above information, the sample source domain image The category labeling result is the k-th category labeling result, and the sample source domain image. The modality annotation result is the i-th modality annotation result, while the sample source domain image The category labeling result is the k-th category labeling result, and the sample source domain image. The modality annotation result is the j-th modality annotation result. Therefore, the sample source domain image Image features and sample source domain images Image features A source domain image feature pair can be used as the labeling result of the k-th category. The distance of this source domain image feature pair can be represented as... The distance between the source image feature pairs is and The square of the L2 norm between them.
[0104] By calculating the feature distance between image features of sample source domain images corresponding to two different modalities labeled with the same category, we can maximize the pairwise similarity between sample source domain images of sample source domain images corresponding to different modalities labeled with the same category. By utilizing complementary knowledge of different modalities, we can mitigate domain shift between modalities and avoid negative transfer. On the other hand, making the sample source domain images of sample source domain images corresponding to different modalities labeled with the same category correspond to similar image features is beneficial for aligning the source domain images and the target domain images, thus facilitating subsequent processing such as modality classification and category classification.
[0105] Based on the distances between the source domain image feature pairs corresponding to any category labeling result, a first loss corresponding to that category labeling result is determined. For example, the distances between the source domain image feature pairs corresponding to any category labeling result are calculated by averaging, summing, etc., and the calculated result is used as the first loss corresponding to that category labeling result.
[0106] Step 205: Based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, train the neural network model to obtain the image classification model.
[0107] The multi-source modality alignment loss can be determined based on the first loss corresponding to each category's annotation results. Optionally, the first losses corresponding to each category's annotation results can be averaged, summed, etc., and the calculated results can be used as the multi-source modality alignment loss. For example, the calculation formula for the multi-source modality alignment loss is shown in the following formula (1).
[0108]
[0109] Among them, L 1 Characterizes the multi-source modal alignment loss. K represents the number of category labeling results. Y s This represents the set of category labels composed of the labeling results for each category. m represents the number of modal labels corresponding to the labeling result of the k-th category. ∑ is the summation symbol. The distance to a source domain image feature pair corresponding to the labeling result of the k-th category is described above and will not be repeated here.
[0110] In this embodiment, the first category prediction results of each sample source domain image and the first category prediction results of each sample target domain image are used to determine the loss. Based on the determined loss and the multi-source modality alignment loss, the loss of the neural network model can be determined, so as to use the loss of the neural network model to train the neural network model and obtain the trained neural network model.
[0111] If the trained neural network model meets the training termination condition, then the trained neural network model is used as the image classification model; if the trained neural network model does not meet the training termination condition, then the trained neural network model is used as the neural network model for the next training, and the neural network model is trained again in the manner described in steps 201 to 205 above, until the image classification model is obtained.
[0112] This application does not limit the conditions for meeting the training termination criteria. For example, the training termination criteria may be met when the number of training iterations reaches a set number, such as 500 iterations. Alternatively, the training termination criteria may be met when the gradient descent of the neural network model's loss falls within a set range.
[0113] In this embodiment, the multi-source modality alignment loss is calculated based on the distance between the source domain image feature pairs corresponding to each category labeling result. By training the neural network model using the multi-source modality alignment loss, the neural network model can narrow the distance between the source domain image feature pairs corresponding to the same category labeling result, making the sample source domain images of each modality labeling result corresponding to the same category labeling result have similar image features. In other words, the sample source domain images of the same category labeling result have similar image features. Therefore, the image features extracted by the image classification model focus more on distinguishing categories and weaken the distinction between modalities, which can improve the classification accuracy when performing image classification based on image features.
[0114] In one possible implementation, step 205 includes steps 2051 to 2053.
[0115] Step 2051: Determine the second loss based on the first category prediction results and the category labeling results of each sample source domain image.
[0116] In this embodiment, the first category prediction result of any sample source domain image includes the probability that the sample source domain image belongs to each category labeling result. The category labeling result of any sample source domain image indicates that the sample source domain image belongs to a certain category labeling result and does not belong to other category labeling results.
[0117] The number of category labeling results can be denoted as N. The first category prediction result for any sample source image is a matrix, denoted as Y. Matrix Y includes N data points, each representing the probability that the sample source image belongs to the category labeling result corresponding to that data point. Each data point is a value greater than or equal to 0 and less than or equal to 1. The category labeling result for any sample source image is also a matrix, denoted as P. Matrix P includes N data points, each representing either the sample source image belonging to the category labeling result corresponding to that data point or not belonging to that category labeling result. Each data point is one of these two values. For example, any data point in matrix P can be 1, representing that the sample source image belongs to the category labeling result corresponding to that data point; any data point in matrix P can also be 0, representing that the sample source image does not belong to the category labeling result corresponding to that data point.
[0118] Optionally, the source domain image includes the labeling results for each category, while the target domain image may include other categories in addition to at least one category labeling result. For example, the source domain image includes two categories: apple and pear, while the target domain image includes three categories: apple, banana, and watermelon. Since banana and watermelon belong to categories not included in the source domain image, they can be considered as other categories.
[0119] Since the embodiments of this application require classifying the images of the target domain, and the images of the target domain may include other categories, for any sample source domain image, the neural network model can determine the probability that the sample source domain image belongs to each category labeling result and the probability that the sample source domain image belongs to other categories, making the transfer mechanism more reliable. Here, the transfer mechanism refers to transferring the category classification of the source domain image to the category classification of the target domain image.
[0120] In this scenario, assuming the number of category labeling results is N, the matrix Y corresponding to the first category prediction result of any sample source image includes N+1 data points. Each of the first N data points represents the probability that the sample source image belongs to the category labeling result corresponding to that data point, and the last data point represents the probability that the sample source image belongs to the unknown category. Similarly, the matrix P corresponding to the category labeling result of any sample source image also includes N+1 data points. Each of the first N data points represents whether the sample source image belongs to the category labeling result corresponding to that data point or not, and the last data point represents whether the sample source image belongs to the unknown category.
[0121] It is understandable that the last data point mentioned above corresponds to an unknown category as one possible implementation. In practice, any data point in matrix Y or matrix P may correspond to an unknown category.
[0122] In this embodiment, the second loss can be calculated based on the first category prediction results and the category labeling results of each sample source domain image according to the second loss function, that is, the second loss is calculated based on matrix Y and matrix P. This embodiment does not limit the second loss function; for example, the second loss function can be the cross-entropy loss function, the relative entropy loss function, etc.
[0123] Step 2052: Determine the third loss based on the first category prediction results of the target domain images of each sample.
[0124] This application embodiment does not require category labeling of the sample target domain images; therefore, there are no category labeling results for the sample target domain images. A third loss can be determined based on the first category prediction results and other information for each sample target domain image. For example, the other information includes at least one of the image features of the sample target domain image, the image features of the sample source domain image, etc.
[0125] In one possible implementation, step 2052 includes steps A1 to A3.
[0126] Step A1: Based on the first category prediction results of each sample target domain image, determine the weight coefficients of each sample target domain image.
[0127] In this embodiment of the application, the first category prediction result of any sample target domain image includes the probability that the sample target domain image belongs to each category labeling result. Therefore, based on the probability that the sample target domain image belongs to each category labeling result, the sum of probabilities or the average probability can be determined, and the determined data can be used as the weight coefficient of the sample target domain image.
[0128] Optionally, the first category prediction result for any sample target domain image also includes the probability that the sample target domain image belongs to other categories. The probability that the sample target domain image belongs to other categories can be used as the weight coefficient of the sample target domain image, or the weight coefficient of the sample target domain image can be calculated based on the probability of the sample target domain image belonging to each category annotation result and the probability that the sample target domain image belongs to other categories.
[0129] Optionally, the first category prediction result of any sample target domain image includes the first probability that the sample target domain image belongs to a category other than the category annotation results. In this case, step A1 includes: obtaining the second category prediction result of each sample target domain image, wherein the second category prediction result of any sample target domain image includes the second probability that the sample target domain image belongs to the category annotation results; and for any sample target domain image, determining the weight coefficient of the sample target domain image based on the first probability and the second probability corresponding to the sample target domain image.
[0130] In this embodiment, the first category prediction result of any sample target domain image may include the second category prediction result of the sample target domain image and the probability that the sample target domain image belongs to other categories. The second category prediction result of the sample target domain image includes the probability that the sample target domain image belongs to each category labeling result; this probability can be denoted as the second probability corresponding to the sample target domain image. The probability that the sample target domain image belongs to other categories can be denoted as the first probability corresponding to the sample target domain image.
[0131] Optionally, the neural network model further includes a second-class classifier, which is used to classify the sample target domain image to obtain a second-class prediction result for the sample target domain image. The second-class classifier is concatenated after the first generator. The first source domain image corresponding to any sample target domain image generated by the first generator, or the image features of the first source domain image, is used as input information for the second-class classifier. The second-class classifier classifies the first source domain image based on the input information to obtain a second-class prediction result for the sample target domain image. This application embodiment does not limit the model structure, model size, etc., of the second-class classifier.
[0132] For example, the second category classifier includes an activation function, which can be any activation function including the Leaky-Softmax function. The activation function can determine the second probability that the sample target domain image belongs to each category labeling result by determining the similarity between the first source domain image corresponding to the sample target domain image and the image in the source domain. Optionally, the sum of the second probabilities that the sample target domain image belongs to each category labeling result is greater than or equal to 0 and less than 1.
[0133] For any sample target domain image, the second probabilities of the sample target domain image belonging to each category annotation result can be added together to obtain the sum of the second probabilities corresponding to the sample target domain image. Optionally, the sum of the second probabilities is determined according to the formula (2) shown below.
[0134]
[0135] Where P(X) represents the sum of the second probabilities corresponding to the sample target domain image. C s The number of category labeling results. The second probability characterizes the target domain image of the sample belonging to the i-th category labeling result.
[0136] Three concepts are introduced in the embodiments of this application. Please refer to... Figure 3 , Figure 3 This is a schematic diagram illustrating the categories between an image in a source domain and an image in a target domain, provided in an embodiment of this application. Categories that the image in the source domain has but the image in the target domain does not have are called source domain private categories; categories that the image in the source domain has and the image in the target domain also have are called public categories; and categories that the image in the source domain does not have but the image in the target domain has are called target domain private categories.
[0137] In this embodiment, the sum of the second probabilities corresponding to the sample target domain image can intuitively reflect the probability that the sample target domain image belongs to the public category, highlighting the similarity between the sample target domain image and the source domain image, and can also indirectly reflect the probability that the sample target domain image belongs to the target domain private category. When P(X) is high, the category of the sample target domain image is more likely to belong to the public category; when P(X) is low, the category of the sample target domain image is more likely to belong to the target domain private category.
[0138] The weight coefficient of a sample target domain image can be determined based on the sum of the first probability and the second probability corresponding to any sample target domain image. For example, the weight coefficient of a sample target domain image is determined according to the following formula (3).
[0139]
[0140] Among them, W t The weight coefficients represent the target domain image of the sample, P(X) represents the sum of the second probabilities corresponding to the target domain image of the sample, and P(y=N+1|x) represents the first probability corresponding to the target domain image of the sample. un (x) represents the third probability corresponding to the target domain image of the sample. N represents the number of category labels, where N+1 corresponds to other categories.
[0141] In this embodiment, the first probability corresponding to the sample target domain image reflects the probability that the sample target domain image belongs to a private category of the target domain. By summing the first probability and the second probability corresponding to the sample target domain image, the possibility of the sample target domain image belonging to a public category or a private category of the target domain can be explored, thereby determining the category of the sample target domain image.
[0142] To make P un If (x) is small, training can be used to increase the first probability that the sample target domain image belongs to the private category of the target domain; to make P un If (x) is large, training can reduce the probability that a sample target domain image belongs to a private category of the target domain. Generally, a sample target domain image belonging to a public category has a higher probability of P compared to a sample target domain image belonging to a private category. un The value of (x) is larger. Therefore, P un (x) is the key parameter that determines whether the sample target domain image belongs to a private category of the target domain. Through P un P(x) and P(X) calculate the weight coefficients of the target domain image of the sample, which can make the target domain image belonging to the public category closer to the source domain image than the target domain image belonging to the private category of the target domain.
[0143] Step A2: Determine the weight coefficients of the source domain images of each sample based on the weight coefficients of the target domain images of each sample.
[0144] In this embodiment, if the category of the sample target domain image belongs to a private category of the target domain, the sum of the second probabilities corresponding to the sample target domain image is approximately equal to 0, and the third probability corresponding to the sample target domain image is also approximately equal to 0. This results in a lower weight coefficient for the sample target domain image, making it more likely that the category of the sample target domain image is determined to be a private category of the target domain (i.e., the sample target domain image belongs to other categories). Conversely, if the category of the sample target domain image belongs to a public category, the sum of the second probabilities corresponding to the sample target domain image is larger, and the third probability corresponding to the sample target domain image is also larger. This results in a higher weight coefficient for the sample target domain image, making it more likely that the category of the sample target domain image is determined to be a public category (i.e., the sample target domain image belongs to any category labeling result).
[0145] Since images of target domains belonging to the public category are closer to images of the source domain than images of target domains belonging to the private category, sample target domain images with larger weight coefficients (hereinafter referred to as reference images) can be selected, and the weight coefficients of any sample source domain image can be determined using the reference images.
[0146] Optionally, step A2 includes: determining multiple reference images from each sample target domain image based on the weight coefficients of each sample target domain image, wherein the weight coefficients of the reference images are greater than a weight threshold; and for any sample source domain image, determining the weight coefficients of any sample source domain image based on the second probability of each reference image belonging to the category labeling result of any sample source domain image.
[0147] If the weight coefficient of any sample target domain image is greater than the weight threshold, then that sample target domain image is used as a reference image, and the second probability of the reference image belonging to each category labeling result is obtained. In this way, multiple reference images can be selected from each sample target domain image, and the second probability of each reference image belonging to each category labeling result can be obtained. This application does not limit the value of the weight threshold; for example, the weight threshold is a hyperparameter and can be set to 0.6 or 0.5.
[0148] Since one sample source domain image corresponds to one category labeling result, the second probability of each reference image belonging to any sample source domain image can be averaged, summed, or otherwise calculated, and the result can be used as the weight coefficient of that sample source domain image. For example, the weight coefficient of the sample source domain image is determined according to the formula (4) shown below.
[0149]
[0150] Where V represents the weight coefficient of each category's annotation result. avg represents the average value. W t (x t >w0) represents the target domain image of the sample x t Weighting coefficient W t When the value is greater than the weight threshold w0, the sample target domain image x t It can be used as a reference image. The second probability representing the reference image belonging to each category of the annotation results. W s (x s Characterizing the source domain image x s The weighting coefficients, Characterizing the yth s The weight coefficients of each category labeling result, and the sample source domain image x s Corresponding to the yth s The results of the category labeling.
[0151] Step A3: Determine the third loss based on the weight coefficients of the target domain images of each sample and the weight coefficients of the source domain images of each sample.
[0152] The third loss can be determined based on the weight coefficients of each sample target domain image, each sample target domain image, each sample source domain image, and each sample source domain image. It is understood that any information obtained after processing the sample target domain images and sample source domain images through the neural network model can participate in the calculation of the third loss. For example, the third loss can be determined based on the weight coefficients of each sample target domain image, the image features of each sample target domain image, the weight coefficients of each sample source domain image, and the image features of each sample source domain image.
[0153] Optionally, step A3 includes: obtaining the domain prediction results of each sample target domain image and the domain prediction results of the first target domain image corresponding to each sample source domain image; and determining the third loss based on the weight coefficients of each sample target domain image, the domain prediction results of each sample target domain image, the weight coefficients of each sample source domain image, and the domain prediction results of the first target domain image corresponding to each sample source domain image.
[0154] In this embodiment, the neural network model further includes a domain classifier. The domain classifier is concatenated after the feature extractor and is used to classify the sample target domain image based on the image features extracted by the feature extractor, thereby obtaining a domain prediction result for the sample target domain image. The domain prediction result includes the probability that the sample target domain image belongs to the source domain and / or the probability that the sample target domain image belongs to the target domain.
[0155] Furthermore, a domain classifier is concatenated after the first generator. After the first generator generates a first target domain image corresponding to a sample source domain image, the classifier performs domain classification on the first target domain image based on the first target domain image or its image features, obtaining a domain prediction result for the first target domain image. This domain prediction result includes the probability that the first target domain image belongs to the source domain and / or the probability that the first target domain image belongs to the target domain.
[0156] Next, the weight coefficients of each sample target domain image and the domain prediction results of each sample target domain image are subjected to weighted averaging and weighted summation to obtain the first calculation result. The weight coefficients of each sample source domain image and the domain prediction results of the corresponding first target domain images are subjected to weighted averaging and weighted summation to obtain the second calculation result. A third loss is determined based on the first and second calculation results; for example, the sum of the first and second calculation results is used as the third loss, or the average of the first and second calculation results is used as the third loss.
[0157] Step 2053: Based on the second loss, the third loss, and the first loss corresponding to each category labeling result, train the neural network model to obtain the image classification model.
[0158] The loss of the neural network model can be obtained by adding the second loss, the third loss, and the first loss corresponding to each category labeling result. Alternatively, the multi-source modality alignment loss can be determined based on the first loss corresponding to each category labeling result, and the multi-source modality alignment loss, the second loss, and the third loss can be added together to obtain the loss of the neural network model. Then, the neural network model is trained using the loss of the neural network model to obtain an image classification model.
[0159] In this embodiment, other losses can also be calculated. The calculated losses are then combined with the second loss, the third loss, and the first loss corresponding to each category labeling result to determine the loss of the neural network model. Several possible implementation methods are provided below, denoted as Implementation Method B1 to Implementation Method B3.
[0160] In implementation B1, step 206 is included after step 205.
[0161] Step 206: Determine the modality prediction results of each sample source domain image through a neural network model; determine the fourth loss based on the modality prediction results and modality annotation results of each sample source domain image.
[0162] The neural network model also includes a modality classifier. The modality classifier is concatenated after the feature extractor and is used to perform modality classification on the source domain image based on the image features extracted by the feature extractor, thus obtaining the modality prediction result of the source domain image. Since each source domain image corresponds to one modality labeling result, there are multiple modality labeling results. In this embodiment, the modality prediction result of the source domain image includes the probability that the source domain image belongs to each modality labeling result.
[0163] The fourth loss can be determined using the modality prediction and modality annotation results of each sample source domain image according to the fourth loss function. For example, the fourth loss function can be the relative entropy loss function, the cross-entropy loss function, etc. The fourth loss can be determined according to the formula (5) shown below.
[0164]
[0165] Among them, L 4 The fourth loss is represented by m, which represents the number of sample source domain images. Characterizing the source domain image of the i-th sample Image features. f m (·) Characterize the modality classifier. Characterizing the source domain image of the i-th sample Modal prediction results. Y s Characterizing the source domain image of the i-th sample Modal annotation results. Characterizing the source domain image of the i-th sample Modal loss.
[0166] Based on the determination of the fourth loss, step 205 includes: training the neural network model based on the fourth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model.
[0167] The second and third losses can be determined based on the first-class prediction results of each sample source domain image and each sample target domain image. The second, third, and fourth losses are then added to the first loss corresponding to each class labeling result to obtain the loss of the neural network model. Alternatively, the multi-source modality alignment loss can be determined based on the first loss corresponding to each class labeling result, and the multi-source modality alignment loss, the second, third, and fourth losses are added to obtain the loss of the neural network model. Finally, the neural network model is trained using the losses of the neural network model to obtain an image classification model.
[0168] In implementation B2, step 207 is included after step 205.
[0169] Step 207: Obtain the domain prediction results of the first target domain image corresponding to each sample source domain image and the domain prediction results of each sample target domain image; determine the fifth loss based on the domain prediction results of the first target domain image corresponding to each sample source domain image and the domain prediction results of each sample target domain image.
[0170] Step A3 has already described how to determine the domain prediction results of the sample target domain image and the domain prediction results of the first target domain image corresponding to the sample source domain image, which will not be repeated here. The fifth loss can be determined according to the fifth loss function, based on the domain prediction results of the first target domain image corresponding to each sample source domain image and the domain prediction results of each sample target domain image.
[0171] For example, the domain loss component corresponding to the source domain is determined based on the domain prediction result of the first target domain image corresponding to each sample source domain image, and the domain loss component corresponding to the target domain is determined based on the domain prediction result of each sample target domain image. A fifth loss is then determined based on the domain loss components corresponding to the source domain and the target domain.
[0172] Alternatively, the fifth loss can be determined according to formula (6) as shown below.
[0173]
[0174] in, Characterizes the fifth loss. The domain loss component corresponding to the source domain is represented. Here, E represents the sign of the average function, log represents the sign of the logarithm, and F(x) ~ D. s The characterization belongs to the source domain D s Image features of the sample source domain image x, The first target domain image representing the source domain image of the sample The domain prediction results. The domain loss component representing the target domain. Where F(x) ~ D t The representation belongs to the target domain D t Image features of the target domain image x, D Y (x) represents the domain prediction result of the sample target domain image.
[0175] Based on the determination of the fifth loss, step 205 includes: training the neural network model based on the fifth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model.
[0176] The second and third losses can be determined based on the first-class prediction results of each sample source domain image and each sample target domain image. The second, third, and fifth losses are then added to the first losses corresponding to each class labeling result to obtain the loss of the neural network model. Alternatively, the multi-source modality alignment loss can be determined based on the first losses corresponding to each class labeling result, and the multi-source modality alignment loss, the second, third, and fifth losses are then added to obtain the loss of the neural network model. Finally, the neural network model is trained using these losses to obtain an image classification model.
[0177] For example, embodiments of this application may also obtain the domain prediction results of the first source domain image corresponding to each sample target domain image and the domain prediction results of each sample source domain image; based on the domain prediction results of the first source domain image corresponding to each sample target domain image and the domain prediction results of each sample source domain image, a seventh loss is determined. The method for determining the seventh loss is similar to the method for determining the fifth loss, and will not be described again here.
[0178] Next, based on the seventh loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, the neural network model is trained to obtain the image classification model. For example, the second loss, third loss, seventh loss, and the first loss corresponding to each category labeling result are added together to obtain the loss of the neural network model. Alternatively, the multi-source modality alignment loss is determined based on the first loss corresponding to each category labeling result, and the multi-source modality alignment loss, second loss, third loss, and seventh loss are added together to obtain the loss of the neural network model. Then, the neural network model is trained using the loss of the neural network model to obtain the image classification model.
[0179] In implementation B3, step 208 is included after step 205.
[0180] Step 208: Generate a second source domain image corresponding to each sample source domain image based on the first target domain image corresponding to each sample source domain image; determine the sixth loss based on each sample source domain image and the second source domain image corresponding to each sample source domain image.
[0181] The neural network model also includes a second generator, which is concatenated after the first generator. The second generator is a generator with a built-in attention mechanism, and the direction of this built-in attention mechanism is opposite to that of the first generator. The second generator can learn the mapping relationship between images in the target domain and images in the source domain. After the first generator generates a first target domain image corresponding to a sample source domain image, based on this first target domain image or its image features, it maps the first target domain image to a second source domain image corresponding to the sample source domain image.
[0182] The sample source domain image is an image of the source domain, while the first target domain image corresponding to this sample source domain image is an image of the target domain. However, the second source domain image corresponding to this sample source domain image is an image of the source domain. Therefore, the first generator and the second generator form a loop structure. After converting the source domain image into the target domain image, it is necessary to convert the target domain image back into the source domain image to preserve the source domain information and avoid mode collapse.
[0183] In other words, for each sample source domain image, the sample source domain image can be reconstructed using the first generator and the second generator. Where x represents the sample source domain image, The first target domain image corresponding to the source domain image x of the sample. The second source domain image corresponding to the sample source domain image x.
[0184] Optionally, generating a second source domain image corresponding to each sample source domain image based on the first target domain image corresponding to each sample source domain image includes: generating source domain features and image content features corresponding to the first target domain image corresponding to any sample source domain image based on the image features of the first target domain image corresponding to any sample source domain image; and generating a second source domain image corresponding to any sample source domain image based on the source domain features and image content features corresponding to the first target domain image corresponding to any sample source domain image.
[0185] The second generator generates a second source domain image corresponding to the first target domain image based on the sample source domain image. Essentially, it generates a source domain image based on an image of the target domain. Therefore, the implementation method can be found in the description of step 2031 above. The implementation principles of the two are similar and will not be repeated here.
[0186] In this embodiment, the sixth loss can be determined based on each sample source domain image and its corresponding second source domain image according to the sixth loss function. This embodiment does not limit the sixth loss function. For example, the distance (e.g., L1 distance, L2 distance, etc.) between any sample source domain image and its corresponding second source domain image is calculated, and this distance is used as the distance corresponding to the sample source domain image. The sixth loss is determined based on the distances corresponding to each sample source domain image. Alternatively, the similarity (e.g., cosine similarity) between the image features of any sample source domain image and the image features of its corresponding second source domain image is calculated, and this similarity is used as the similarity corresponding to the sample source domain image. The sixth loss is determined based on the similarity corresponding to each sample source domain image.
[0187] Based on the determination of the sixth loss, step 205 includes: training the neural network model based on the sixth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model.
[0188] The second and third losses can be determined based on the first-class prediction results of each sample source domain image and each sample target domain image. The second, third, and sixth losses, along with the first losses corresponding to each class labeling result, are added to obtain the loss of the neural network model. Alternatively, the multi-source modality alignment loss can be determined based on the first losses corresponding to each class labeling result, and the multi-source modality alignment loss, the second, third, and sixth losses are added to obtain the loss of the neural network model. Then, the neural network model is trained using the losses of the neural network model to obtain an image classification model.
[0189] For example, embodiments of this application may also generate a second target domain image corresponding to each sample target domain image based on the first source domain image corresponding to each sample target domain image; and determine an eighth loss based on each sample target domain image and the second target domain image corresponding to each sample target domain image.
[0190] After the first generator generates a first source domain image corresponding to the sample target domain image, the second generator can map the first source domain image to a second target domain image corresponding to the sample target domain image based on the first source domain image or the image features of the first source domain image.
[0191] The sample target domain image is an image of the target domain, while the corresponding first source domain image is an image of the source domain. However, the corresponding second target domain image is also an image of the target domain. Therefore, the first generator and the second generator form a loop structure. After converting the target domain image into a source domain image, it is necessary to convert the source domain image back into a target domain image to preserve the target domain information and avoid mode collapse.
[0192] In other words, for each sample target domain image, the target domain image can be reconstructed using the first generator and the second generator. Where x represents the target domain image of the sample, The first source domain image corresponding to the target domain image x of the sample. The second target domain image corresponding to the target domain image x of the sample.
[0193] Optionally, generating a second target domain image corresponding to each sample target domain image based on the first source domain image corresponding to each sample target domain image includes: generating target domain features and image content features corresponding to the first source domain image corresponding to any sample target domain image based on the image features of the first source domain image corresponding to any sample target domain image; and generating a second target domain image corresponding to any sample target domain image based on the target domain features and image content features corresponding to the first source domain image corresponding to any sample target domain image.
[0194] The second generator generates a second target domain image corresponding to the first source domain image corresponding to the sample target domain image. Essentially, it generates a target domain image based on an image of the source domain. Therefore, the implementation method can be found in the description of step 2021 above. The implementation principles of the two are similar and will not be repeated here.
[0195] In this embodiment, the eighth loss can be determined based on the target domain image of each sample and the corresponding second target domain image, according to the eighth loss function. This embodiment does not limit the eighth loss function. For example, the distance (e.g., L1 distance, L2 distance, etc.) between any target domain image and the corresponding second target domain image is calculated, and this distance is used as the distance corresponding to the target domain image. The eighth loss is determined based on the distances corresponding to each target domain image. Alternatively, the similarity (e.g., cosine similarity) between the image features of any target domain image and the image features of the corresponding second target domain image is calculated, and this similarity is used as the similarity corresponding to the target domain image. The eighth loss is determined based on the similarity corresponding to each target domain image.
[0196] Based on the determined eighth loss, the neural network model is trained using the eighth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain the image classification model.
[0197] It should be noted that the embodiments of this application can train the neural network model to obtain the image classification model based on at least one of the multi-source modality alignment loss and the second to eighth losses. In the embodiments of this application, a pre-trained residual network can be used as a feature extractor, and during pre-training, the initial learning rate of all network layers in the feature extractor except the last two layers is set to 10. -6 The initial learning rate for the last two network layers in the feature extractor is 10. -5 Set the learning rate decay rate to 0.5 every 1000 steps. Build a neural network model using the pre-trained feature extractor and train it. An optimizer (such as the Adam optimizer) can be used for training. The optimizer's two hyperparameters are beta1 = 0.5 and beta2 = 0.999, with a weight decay of 0.0001. Adjust the model parameters based on the hyperparameter λ = 0.1. The neural network model is trained for a total of 10,000 iterations. If the neural network model converges prematurely, stop training.
[0198] During each training iteration, a total of 256 source and target domain images are used. Data augmentation can be performed on the source and / or target domain images, for example, by performing affine transformations on the target domain images to increase the number of images.
[0199] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample source domain images and sample target domain images involved in this application were obtained with full authorization.
[0200] The above method, for any category labeling result, uses the image features of the source domain images of two different modal labeling results corresponding to that category labeling result as a source domain image feature pair. This is beneficial for comparing the image features of the source domain images of two sample source domain images with the same category labeling result but different modal labeling results. The first loss corresponding to that category labeling result is determined using each source domain image feature pair, and the image classification model is trained using this first loss. This allows the image classification model to narrow down the image features of sample source domain images of two sample source domain images with the same category labeling result but different modal labeling results, making the image features more focused on distinguishing categories and less focused on distinguishing modalities. Therefore, the image features extracted by the image classification model are more conducive to image classification, improving the accuracy of the classification results.
[0201] This application also provides an image classification method, which can be applied to the above-described implementation environment. Figure 4 The flowchart shown in this embodiment of the application illustrates an image classification method. This method can be implemented by... Figure 1 The method can be executed by either terminal device 101 or server 102, or jointly by both. For ease of description, the terminal device 101 or server 102 executing the image classification method in this embodiment is referred to as an electronic device, and the method can be executed by an electronic device. Figure 4 As shown, the method includes the following steps.
[0202] Step 401: Obtain the image to be classified, which can be either a source domain image or a target domain image.
[0203] The image to be classified can be an image of the source domain (i.e., a source domain image) or an image of the target domain (i.e., a target domain image). This application does not limit the method of acquiring the image to be classified. For example, the electronic device can receive the image to be classified sent by other devices or manually input, or the electronic device can capture the image to be classified.
[0204] Step 402: Determine the image features of the image to be classified using an image classification model.
[0205] The image classification model was trained using the same training method as the image classification model related to weight 2.
[0206] Image classification models include feature extractors, which are used to extract features from the image to be classified, thus obtaining the image features of the image to be classified. The method for determining the image features of the image to be classified can be found in the description of image features of the sample source domain image above; the implementation principles are similar and will not be repeated here.
[0207] Step 403: Determine the category of the image to be classified based on its image features using an image classification model.
[0208] In this embodiment, an image classification model can determine the category prediction result of the image to be classified based on its image features. The process of determining the category prediction result is described in steps 202 and 203, which are similar in principle and will not be repeated here.
[0209] Next, the category of the image to be classified is determined based on the category prediction results. Optionally, the category prediction results of the image to be classified include the probability that the image belongs to each category label. If the probability that the image belongs to any category label is greater than or equal to the reference probability, then the category of the image to be classified is determined to be that category label.
[0210] This application does not limit the value of the reference probability in its embodiments. For example, the reference probability is the highest probability among the probabilities of the image to be classified belonging to each category labeling result, or the probabilities of the image to be classified belonging to each category labeling result are sorted, and the predetermined number of probabilities after sorting is used as the reference probability, or the reference probability is a set data.
[0211] Optionally, the category prediction result of the image to be classified includes the probability that the image belongs to each category label and the probability that the image belongs to other categories besides the category labels. If the probability that the image belongs to any category label is greater than or equal to the reference probability, then the category of the image to be classified is determined to be that category label.
[0212] In one possible implementation, step 403 includes steps 4031 to 4032.
[0213] Step 4031: When the image to be classified is a source domain image, the target domain image corresponding to the source domain image is generated based on the image features of the source domain image using an image classification model; the category of the source domain image is determined based on the target domain image corresponding to the source domain image using an image classification model.
[0214] Step 4031 requires determining the category prediction result of the source domain image based on the target domain image corresponding to the source domain image, and then determining the category of the source domain image based on the category prediction result of the source domain image. The details of determining the category prediction result of the source domain image are described in steps 2021 and 2022, both of which are processes for determining the category prediction result of the source domain image and have the same implementation principle, so they will not be repeated here.
[0215] Optionally, in step 4031, generating the target domain image corresponding to the source domain image based on the image features of the source domain image using an image classification model includes: generating target domain features and image content features corresponding to the source domain image based on the image features of the source domain image using an image classification model; and generating the target domain image corresponding to the source domain image based on the target domain features and image content features corresponding to the source domain image using an image classification model.
[0216] The process of generating the target domain image corresponding to the source domain image can be seen in the description of step 2021. Step 2021 involves the process of generating the first target domain image corresponding to the sample source domain image. The two are implemented in the same way and will not be described again here.
[0217] Step 4032: When the image to be classified is a target domain image, the source domain image corresponding to the target domain image is generated based on the image features of the target domain image using an image classification model; the category of the target domain image is determined based on the source domain image corresponding to the target domain image using an image classification model.
[0218] Step 4032 requires determining the category prediction result of the target domain image based on the source domain image corresponding to the target domain image, and then determining the category of the target domain image based on the category prediction result. The details of determining the category prediction result of the target domain image can be found in the descriptions of steps 2031 and 2032, both of which are processes for determining the category prediction result of the target domain image and have the same implementation principle, so they will not be repeated here.
[0219] Optionally, in step 4032, generating the source domain image corresponding to the target domain image based on the image features of the target domain image using an image classification model includes: generating source domain features and image content features corresponding to the target domain image based on the image features of the target domain image using an image classification model; and generating the source domain image corresponding to the target domain image based on the source domain features and image content features corresponding to the target domain image using an image classification model.
[0220] The process of generating the source domain image corresponding to the target domain image can be seen in the description of step 2031. Step 2031 involves the process of generating the first source domain image corresponding to the sample target domain image. The two are implemented in the same way and will not be described again here.
[0221] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the images to be classified involved in this application were obtained with full authorization.
[0222] The image features extracted by the image classification model described above are more conducive to image classification. Therefore, when using this image classification model to classify images, the classification accuracy is high. Furthermore, this image classification model can be applied to various classification scenarios, such as classifying medical images to achieve purposes like diagnosing diseases and analyzing lesion types.
[0223] The foregoing section described the training method and image classification method of the image classification model provided in the embodiments of this application from the perspective of method and steps. The following section systematically describes the training process of the image classification model provided in the embodiments of this application. Please refer to... Figure 5 , Figure 5 (1) is a training framework diagram of the image classification model provided in the embodiments of this application.
[0224] In this embodiment, a neural network model is trained using sample source domain images and sample target domain images to obtain an image classification model. The neural network model includes a feature extractor, a modality classifier, a domain classifier, a first generator, a second generator, a first category classifier, a second category classifier, and a discriminator. The feature extractor extracts image features from both the sample source domain images and the sample target domain images. The image features from the sample source domain images are used to determine the first loss corresponding to each category labeling result, thereby determining the multi-source modality alignment loss.
[0225] The image features of the sample source domain image can also be input into the first generator, which generates a first target domain image corresponding to the sample source domain image based on the image features. The first target domain image can be input into the first classifier, which determines the first class prediction result of the sample source domain image. The first class prediction result of the sample source domain image is used together with the class labeling result of the sample source image to determine the second loss.
[0226] The first target domain image corresponding to the sample source domain image can also be input into a second classifier. The second classifier determines the second class prediction result of the sample source domain image. The first and second class prediction results are then input into a discriminator, which generates weight coefficients for the sample target domain image and determines the weight coefficients for the sample source domain image based on these weight coefficients. Image features of the sample target domain image can be input into a domain classifier, which outputs the domain prediction result for the sample target domain image. The first target domain image can also be input into a domain classifier, which outputs the domain prediction result for the first target domain image. The weight coefficients of the sample target domain image, the domain prediction result of the sample target domain image, the weight coefficients of the sample source domain image, and the domain prediction result of the first target domain image are used to determine the third loss.
[0227] Image features from the source domain image are input into a modality classifier to perform modality classification on the source domain image, yielding modality prediction results. These modality prediction results, along with the modality annotation results of the source domain image, are used to determine the fourth loss.
[0228] The domain prediction results of the first target domain image and the domain prediction results of the sample target domain image are also used to determine the fifth loss.
[0229] The first target domain image can also be input into a second generator, which generates a second source domain image corresponding to the sample source domain image. The sample source domain image and the second source domain image are used to determine the sixth loss.
[0230] Next, the multi-source modal alignment loss and the second through sixth losses can be added together to obtain the loss of the neural network model. The neural network model is then trained using this loss to obtain an image classification model.
[0231] The weight coefficients of the sample target domain image can be used to adjust the first-class classifier. See also... Figure 5 (2) in the middle Figure 5 (2) is a schematic diagram illustrating the influence of weight coefficients on the category of a sample target domain image provided in an embodiment of this application. In this embodiment, the sample target domain image is input into a feature extractor, which extracts image features of the sample target domain image. Based on the image features of the sample target domain image, a first category prediction result of the sample target domain image can be determined by a first category classifier; based on the image features of the sample target domain image, a discriminator can determine the weight coefficients of the sample target domain image, wherein the weight coefficients of the sample target domain image need to be determined based on the first category prediction result of the sample target domain image.
[0232] When the weight coefficient of the sample target domain image is less than the weight threshold, it can be determined that the sample target domain image likely belongs to the private category of the target domain based on the first category prediction result of the sample target domain image; when the weight coefficient of the sample target domain image is not less than the weight threshold, it can be determined that the sample target domain image likely belongs to the public category based on the first category prediction result of the sample target domain image.
[0233] After training an image classification model, it can be used to classify the images to be classified. See also... Figure 6 , Figure 6 This is a schematic diagram illustrating an image classification method provided in an embodiment of this application. In this embodiment, the backend is configured with the image classification model of this application. The backend can receive the image to be classified input from the frontend A, classify the image into categories, and output the category of the image to the frontend B. Optionally, frontend A and frontend B are terminal devices, and the backend is a server.
[0234] Optionally, the image classification model configured in the backend can be the model remaining after removing at least one of the modality classifier, domain classifier, second generator, second class classifier, and discriminator. This saves on the computational and storage requirements of the backend.
[0235] In this embodiment, five sample datasets can be obtained. Each dataset lacks prior knowledge of class overlap between the source and target domain images, and it is uncertain whether the source domain image has multiple modalities. These five datasets are designated as Dataset 1 to Dataset 6. Dataset 1 contains 31 classes across 3 domains, any one of which can be used as either the source or target domain. Dataset 2 contains 65 classes across 4 domains, any one of which can be used as either the source or target domain. Dataset 3 contains 12 classes across 2 domains, which serve as both the source and target domains. Dataset 4 contains 256 classes across 1000 domains, any one of which can be used as either the source or target domain. Dataset 5 contains 345 classes across 6 domains, any one of which can be used as either the source or target domain.
[0236] This application embodiment tested the accuracy of 18 models on datasets 1, 2, and 5. These 18 models are denoted as M1 to M18, where M1 to M17 are image classification models in related technologies, and M18 is the image classification model in this embodiment. The optimizer parameter β = 0.32 was used when training M2 and M3. The optimizer parameter β = 0.15 was used when training M4 to M6. The optimizer parameter β = 0.5 was used when training M7 to M8. The optimizer parameter β = 0.07 was used when training M9 to M14. The optimizer parameter β = 0.43 was used when training M15 and M17. The parameters of the above five optimizers are denoted as i, ii, iii, iv, and v, respectively. Two domains were randomly combined from three domains in dataset 1 to obtain D1 to D3. Two domains were randomly combined from four domains in dataset 2 to obtain D1 to D4. Two domains were randomly combined from six domains in dataset 5 to obtain D1 to D6. Based on the above, the data shown in Table 1 below can be obtained, where E represents the average value.
[0237] Table 1
[0238]
[0239]
[0240]
[0241] As can be seen from Table 1, the data corresponding to M18 is greater than the data corresponding to M1 to M17. Therefore, the image classification model in this embodiment has higher accuracy.
[0242] This application also tested the classification accuracy of 10 models (M1 to M4, M6 to M8, M12 to M13, and M18) in datasets 1, 2, 3, and 5, respectively. Specifically, in dataset 1, two domains were randomly combined from three domains to obtain D1 to D3. In dataset 2, two domains were randomly combined from four domains to obtain D1 to D6. In dataset 3, two domains were randomly combined from two domains to obtain D1 to D2. In dataset 5, two domains were randomly combined from six domains to obtain D1 to D3. Based on the above, the data shown in Table 2 below can be obtained.
[0243] Table 2
[0244]
[0245]
[0246] As shown in Table 2, the classification accuracy of M18 is higher than that of M1 to M4, M6 to M8, and M12. On dataset 3, the classification accuracy of M13 is better than that of M18. However, on datasets 1 and 2, the difference in classification accuracy between M18 and M13 is small. On dataset 5, however, the classification accuracy of M18 is better than that of M13. This is because dataset 3 has only two domains, while datasets 1 and 2 have more domains than dataset 3, and dataset 5 has more domains than both datasets 1 and 2. Therefore, it can be concluded that the image classification model in this embodiment has similar performance to image classification models in related technologies with fewer domains, but in datasets with a larger number of domains, the image classification model in this embodiment can improve classification accuracy.
[0247] In this embodiment of the application, the following will be adopted: Figure 5 The image classification model trained in this way is denoted as Model 1. Furthermore, in... Figure 5 Based on the training process shown, the image classification model trained after removing the modality classifier is denoted as Model 2; Figure 5 Based on the training process shown, the image classification model trained after removing the first and second generators is denoted as Model 3; Figure 5 Based on the training process shown, the image classification model obtained after removing the second classifier and discriminator is denoted as Model 4.
[0248] The performance of models 1 through 4 was tested on datasets 1, 2, and 5. In dataset 1, two domains were randomly combined from three domains to obtain D1. In dataset 2, two domains were randomly combined from four domains to obtain D1 through D3. In dataset 5, two domains were randomly combined from six domains to obtain D1. Based on the above, the data shown in Table 3 below can be statistically analyzed.
[0249] Table 3
[0250]
[0251]
[0252] As shown in Table 3, Model 1 performs better than Models 2, 3, and 4. Therefore, based on... Figure 5 The training process shown yields an image classification model with high classification performance.
[0253] Figure 7 The diagram shown is a structural schematic of a training device for an image classification model provided in an embodiment of this application. Figure 7 As shown, the device includes:
[0254] The acquisition module 701 is used to acquire multiple sample source domain images, multiple sample target domain images, modality annotation results of each sample source domain image, and category annotation results of each sample source domain image;
[0255] The determination module 702 is used to determine the first category prediction result of each sample source domain image through a neural network model;
[0256] The determination module 702 is also used to determine the first category prediction result of the target domain image of each sample through a neural network model;
[0257] The determination module 702 is further configured to, for any category labeling result, determine the first loss corresponding to any category labeling result based on each source domain image feature pair corresponding to any category labeling result, wherein a source domain image feature pair corresponding to any category labeling result includes the image features of the sample source domain images of two different modal labeling results corresponding to any category labeling result;
[0258] Training module 703 is used to train the neural network model based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, to obtain the image classification model.
[0259] In one possible implementation, the determining module 702 is used to generate a first target domain image corresponding to any sample source domain image based on the image features of any sample source domain image using a neural network model; and to determine the first category prediction result of any sample source domain image based on the first target domain image corresponding to any sample source domain image using a neural network model.
[0260] In one possible implementation, the determining module 702 is used to generate target domain features and image content features corresponding to any sample source domain image based on the image features of any sample source domain image; and to generate a first target domain image corresponding to any sample source domain image based on the target domain features and image content features corresponding to any sample source domain image.
[0261] In one possible implementation, the determining module 702 is used to generate a first source domain image corresponding to any sample target domain image based on the image features of any sample target domain image using a neural network model; and to determine the first category prediction result of any sample target domain image based on the first source domain image corresponding to any sample target domain image using a neural network model.
[0262] In one possible implementation, the training module 703 is used to determine a second loss based on the first category prediction results of each sample source domain image and the category labeling results of each sample source domain image; determine a third loss based on the first category prediction results of each sample target domain image; and train the neural network model based on the second loss, the third loss and the first loss corresponding to each category labeling result to obtain an image classification model.
[0263] In one possible implementation, the training module 703 is used to determine the weight coefficients of each sample target domain image based on the first category prediction results of each sample target domain image; determine the weight coefficients of each sample source domain image based on the weight coefficients of each sample target domain image; and determine the third loss based on the weight coefficients of each sample target domain image and the weight coefficients of each sample source domain image.
[0264] In one possible implementation, the first category prediction result of any sample target domain image includes the first probability that any sample target domain image belongs to a category other than the category annotation results;
[0265] The training module 703 is used to obtain the second category prediction result of each sample target domain image. The second category prediction result of any sample target domain image includes the second probability that the sample target domain image belongs to each category label result. For any sample target domain image, the weight coefficient of the sample target domain image is determined based on the first probability and the second probability corresponding to the sample target domain image.
[0266] In one possible implementation, the training module 703 is used to determine multiple reference images from each sample target domain image based on the weight coefficients of each sample target domain image, wherein the weight coefficients of the reference images are greater than a weight threshold; and for any sample source domain image, the weight coefficients of any sample source domain image are determined based on the second probability of each reference image belonging to the category labeling result of any sample source domain image.
[0267] In one possible implementation, the training module 703 is used to obtain the domain prediction results of the target domain images of each sample and the domain prediction results of the first target domain images corresponding to each sample source domain image; and to determine the third loss based on the weight coefficients of the target domain images of each sample, the domain prediction results of the target domain images of each sample, the weight coefficients of the source domain images of each sample, and the domain prediction results of the first target domain images corresponding to each sample source domain image.
[0268] In one possible implementation, the device further includes:
[0269] The determination module 702 is also used to determine the modality prediction results of each sample source domain image through a neural network model; and to determine the fourth loss based on the modality prediction results and modality annotation results of each sample source domain image.
[0270] Training module 703 is used to train the neural network model based on the fourth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model.
[0271] In one possible implementation, the device further includes:
[0272] The determination module 702 is further configured to obtain the domain prediction results of the first target domain image corresponding to each sample source domain image and the domain prediction results of each sample target domain image; and determine the fifth loss based on the domain prediction results of the first target domain image corresponding to each sample source domain image and the domain prediction results of each sample target domain image.
[0273] Training module 703 is used to train the neural network model based on the fifth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model.
[0274] In one possible implementation, the device further includes:
[0275] The determination module 702 is further configured to generate a second source domain image corresponding to each sample source domain image based on the first target domain image corresponding to each sample source domain image; and determine a sixth loss based on each sample source domain image and the second source domain image corresponding to each sample source image.
[0276] Training module 703 is used to train the neural network model based on the sixth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model.
[0277] For any given category labeling result, the aforementioned device uses the image features of the source domain images of two different modalities corresponding to that category labeling result as a source domain image feature pair. This facilitates the comparison of image features of the source domain images of two sample source domain images with the same category labeling result but different modalities. A first loss corresponding to that category labeling result is determined using each source domain image feature pair, and the image classification model is trained using this first loss. This allows the image classification model to narrow down the image features of sample source domain images with the same category labeling result but different modalities, resulting in similar image features for sample source domain images with the same category labeling result but different modalities. This makes the image features more focused on distinguishing categories and less focused on distinguishing modalities. Therefore, the image features extracted by the image classification model are more conducive to image classification, improving the accuracy of the classification results.
[0278] It should be understood that the above Figure 7 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.
[0279] Figure 8 The diagram shown is a structural schematic of an image classification device provided in an embodiment of this application. Figure 8 As shown, the device includes:
[0280] The acquisition module 801 is used to acquire the image to be classified, which is either a source domain image or a target domain image.
[0281] The determination module 802 is used to determine the image features of the image to be classified through an image classification model, wherein the image classification model is trained according to the training method of any of the above-mentioned image classification models;
[0282] The determination module 802 is also used to determine the category of the image to be classified based on the image features of the image to be classified using an image classification model.
[0283] In one possible implementation, the determining module 802 is used to: generate a target domain image corresponding to the source domain image based on the image features of the source domain image using an image classification model when the image to be classified is a source domain image; determine the category of the source domain image based on the target domain image corresponding to the source domain image using an image classification model; and generate a source domain image corresponding to the target domain image based on the image features of the target domain image using an image classification model when the image to be classified is a target domain image; and determine the category of the target domain image based on the source domain image corresponding to the target domain image using an image classification model.
[0284] In one possible implementation, the determining module 802 is used to generate target domain features and image content features corresponding to the source domain image based on the image features of the source domain image using an image classification model; and to generate a target domain image corresponding to the source domain image based on the target domain features and image content features corresponding to the source domain image using an image classification model.
[0285] The determination module 802 is used to generate source domain features and image content features corresponding to the target domain image based on the image features of the target domain image through an image classification model; and to generate source domain images corresponding to the target domain image based on the source domain features and image content features of the target domain image through an image classification model.
[0286] The image features extracted by the image classification model in the above device are more conducive to image classification. Therefore, when using this image classification model to classify images, the classification accuracy is high.
[0287] It should be understood that the above Figure 8 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.
[0288] Figure 9 A structural block diagram of a terminal device 900 provided in an exemplary embodiment of this application is shown. The terminal device 900 includes a processor 901 and a memory 902.
[0289] Processor 901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.
[0290] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one computer program, which is executed by the processor 901 to implement the training method or image classification method of the image classification model provided in the method embodiments of this application.
[0291] In some embodiments, the terminal device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.
[0292] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.
[0293] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 904 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.
[0294] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, disposed on the front panel of terminal device 900; in other embodiments, there may be at least two display screens 905, disposed on different surfaces of terminal device 900 or in a folded design; in other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of terminal device 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).
[0295] The camera assembly 906 is used to acquire images or videos. Optionally, the camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.
[0296] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 901 for processing, or to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal device 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.
[0297] Power supply 908 is used to supply power to the various components in terminal device 900. Power supply 908 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.
[0298] In some embodiments, the terminal device 900 further includes one or more sensors 909. The one or more sensors 909 include, but are not limited to, an accelerometer 911, a gyroscope 912, a pressure sensor 913, an optical sensor 914, and a proximity sensor 915.
[0299] Accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal device 900. For example, accelerometer 911 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 901 can control display screen 905 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 911. Accelerometer 911 can also be used for games or for acquiring user motion data.
[0300] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal device 900. The gyroscope sensor 912, in conjunction with the accelerometer sensor 911, can collect the user's 3D movements on the terminal device 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.
[0301] The pressure sensor 913 can be disposed on the side bezel of the terminal device 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is disposed on the side bezel of the terminal device 900, it can detect the user's grip signal on the terminal device 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is disposed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.
[0302] An optical sensor 914 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity collected by the optical sensor 914. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 914.
[0303] The proximity sensor 915, also known as a distance sensor, is typically located on the front panel of the terminal device 900. The proximity sensor 915 is used to detect the distance between the user and the front of the terminal device 900. In one embodiment, when the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from a screen-off state to a screen-on state.
[0304] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the terminal device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.
[0305] Figure 10This is a schematic diagram of the server structure provided in the embodiments of this application. The server 1000 can vary considerably due to different configurations or performance. It may include one or more processors 1001 and one or more memories 1002. The one or more memories 1002 store at least one computer program, which is loaded and executed by the one or more processors 1001 to implement the image classification model training method or image classification method provided in the above-described method embodiments. For example, the processor 1001 is a CPU. Of course, the server 1000 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1000 may also include other components for implementing device functions, which will not be elaborated here.
[0306] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement the training method or image classification method of any of the above-described image classification models.
[0307] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.
[0308] In an exemplary embodiment, a computer program or computer program product is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement the training method or image classification method of any of the above-described image classification models.
[0309] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0310] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0311] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.
Claims
1. A training method for an image classification model, characterized in that, The method includes: Obtain multiple source domain images, multiple target domain images, modality annotation results for each source domain image, and category annotation results for each source domain image; The first category prediction result of each sample source domain image is determined by a neural network model; The neural network model is used to determine the first category prediction result for each sample target domain image; For any category labeling result, based on each source domain image feature pair corresponding to the any category labeling result, a first loss corresponding to the any category labeling result is determined. The source domain image feature pair corresponding to the any category labeling result includes the image features of the sample source domain images of the two different modal labeling results corresponding to the any category labeling result. Based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, the neural network model is trained to obtain an image classification model.
2. The method according to claim 1, characterized in that, The step of determining the first category prediction result of each sample source domain image through a neural network model includes: For any sample source domain image, the neural network model generates a first target domain image corresponding to the sample source domain image based on the image features of the sample source domain image; The neural network model determines the first category prediction result of any sample source domain image based on the first target domain image corresponding to any sample source domain image.
3. The method according to claim 2, characterized in that, The step of generating a first target domain image corresponding to any one of the sample source domain images based on the image features of that sample source domain image includes: Based on the image features of any one of the sample source domain images, generate the target domain features and image content features corresponding to the sample source domain image; Based on the target domain features and image content features corresponding to any one of the sample source domain images, a first target domain image corresponding to any one of the sample source domain images is generated.
4. The method according to claim 1, characterized in that, The neural network model is trained based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model, including: Based on the first category prediction result and the category labeling result of each sample source domain image, a second loss is determined; Based on the first category prediction results of the target domain images of each sample, a third loss is determined; The neural network model is trained based on the second loss, the third loss, and the first loss corresponding to each category labeling result to obtain an image classification model.
5. The method according to claim 4, characterized in that, The third loss is determined based on the first category prediction results of the target domain images of each sample, including: Based on the first category prediction results of each sample target domain image, the weight coefficients of each sample target domain image are determined; Based on the weight coefficients of the target domain images of each sample, the weight coefficients of the source domain images of each sample are determined. The third loss is determined based on the weight coefficients of the target domain images of each sample and the weight coefficients of the source domain images of each sample.
6. The method according to claim 5, characterized in that, The first category prediction result for any sample target domain image includes the first probability that the sample target domain image belongs to a category other than the category annotation results; The determination of the weight coefficients for each sample target domain image based on the first category prediction results of each sample target domain image includes: Obtain the second category prediction result of each sample target domain image, wherein the second category prediction result of any sample target domain image includes the second probability that the sample target domain image belongs to each category labeling result; For any given sample target domain image, the weight coefficient of the given sample target domain image is determined based on the first probability and the second probability corresponding to the given sample target domain image.
7. The method according to claim 5, characterized in that, Determining the weight coefficients of each sample source domain image based on the weight coefficients of each sample target domain image includes: Based on the weight coefficients of each sample target domain image, multiple reference images are determined from each sample target domain image, wherein the weight coefficients of the reference images are greater than a weight threshold. For any sample source domain image, the weight coefficient of the sample source domain image is determined based on the second probability of the category labeling results of each reference image belonging to the sample source domain image.
8. The method according to claim 5, characterized in that, The determination of the third loss based on the weight coefficients of the target domain images of each sample and the weight coefficients of the source domain images of each sample includes: Obtain the domain prediction results of each sample target domain image and the domain prediction results of the first target domain image corresponding to each sample source domain image. The domain prediction results of the sample target domain image include the probability that the sample target domain image belongs to the source domain and / or the probability that the sample target domain image belongs to the target domain. The domain prediction results of the first target domain image corresponding to the sample source domain image include the probability that the first target domain image corresponding to the sample source domain image belongs to the source domain and / or the probability that the first target domain image corresponding to the sample source domain image belongs to the target domain. The third loss is determined based on the weight coefficients of the target domain images of each sample, the domain prediction results of the target domain images of each sample, the weight coefficients of the source domain images of each sample, and the domain prediction results of the first target domain images corresponding to the source domain images of each sample.
9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: The modal prediction results of each sample source domain image are determined by the neural network model. The fourth loss is determined based on the modality prediction results and modality annotation results of each sample source domain image; The neural network model is trained based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model, including: Based on the fourth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, the neural network model is trained to obtain an image classification model.
10. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Obtain the domain prediction results of the first target domain image corresponding to each sample source domain image and the domain prediction results of each sample target domain image. The domain prediction results of the sample target domain image include the probability that the sample target domain image belongs to the source domain and / or the probability that the sample target domain image belongs to the target domain. The domain prediction results of the first target domain image corresponding to the sample source domain image include the probability that the first target domain image corresponding to the sample source domain image belongs to the source domain and / or the probability that the first target domain image corresponding to the sample source domain image belongs to the target domain. Based on the domain prediction results of the first target domain image corresponding to each sample source domain image and the domain prediction results of each sample target domain image, the fifth loss is determined; The neural network model is trained based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model, including: Based on the fifth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, the neural network model is trained to obtain an image classification model.
11. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Based on the first target domain image corresponding to each sample source domain image, a second source domain image corresponding to each sample source domain image is generated; Based on each sample source domain image and the corresponding second source domain image, a sixth loss is determined; The neural network model is trained based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image to obtain an image classification model, including: Based on the sixth loss, the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, the neural network model is trained to obtain an image classification model.
12. A training device for an image classification model, characterized in that, The device includes: The acquisition module is used to acquire multiple sample source domain images, multiple sample target domain images, modality annotation results of each sample source domain image, and category annotation results of each sample source domain image; A determination module is used to determine the first category prediction result of each sample source domain image through a neural network model; The determining module is further configured to determine the first category prediction result of each sample target domain image through the neural network model; The determining module is further configured to, for any category labeling result, determine a first loss corresponding to the any category labeling result based on each source domain image feature pair corresponding to the any category labeling result, wherein a source domain image feature pair corresponding to the any category labeling result includes image features of sample source domain images of two different modal labeling results corresponding to the any category labeling result; The training module is used to train the neural network model based on the first loss corresponding to each category labeling result, the first category prediction result of each sample source domain image, and the first category prediction result of each sample target domain image, to obtain an image classification model.
13. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the training method of the image classification model as described in any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement the training method of the image classification model as described in any one of claims 1 to 11.
15. A computer program product, characterized in that, The computer program product stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to implement the training method of the image classification model as described in any one of claims 1 to 11.
Citation Information
Patent Citations
Image processing model training method and device, equipment and storage medium
CN114445670A
Model training method, related device, equipment and storage medium
CN114445679A
