Method, device, equipment and storage medium for training encoder

Abstract

This application discloses a method, apparatus, device, and storage medium for training an encoder, belonging to the field of image processing. The method comprises: obtaining a first sample image and a second sample image belonging to the same image category; inputting the first sample image and the second sample image into an encoder, respectively, to obtain a first hash code corresponding to the first sample image and a second hash code corresponding to the second sample image; determining a contrast loss between the first hash code and the second hash code; and training the encoder based on the contrast loss. The encoder training method provided in this application can improve the encoder's accuracy in image classification and retrieval.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, apparatus, device and storage medium for training an encoder. Background Art

[0002] The image hashing method is a common image retrieval method in the field of image retrieval. That is, images can be encoded into hash codes respectively. Then, when retrieving images, the matching degree of the corresponding hash codes of the images is used to determine whether the contents displayed by each image are similar.

[0003] In related technologies, a hash code corresponding to an image can be obtained using a trained generative model. When training a generative model, which may include an encoder and a decoder, the corresponding training process is as follows: a sample image is input into the encoder to obtain a hash code corresponding to the sample image. The hash code corresponding to the sample image is then input into the decoder, which restores the image corresponding to the hash code. Finally, the decoder determines the corresponding loss value between the image restored by the decoder and the sample image, and the loss value is used to train the encoder and decoder.

[0004] Because the generative model converges only when the decoder can accurately reproduce the sample image during training, the trained encoder must output a hash code for the sample image that contains as much information as possible about the corresponding image content. However, when classifying or retrieving images, it is often sufficient to base the classification on a single feature within the image. For example, consider two images of the same car—one taken in an underground parking lot and the other outdoors. Although the backgrounds differ due to the different locations, both images belong to the "car" category. If the hash codes output by the encoder for these two images include a large amount of background information, the hash codes for the two images will have a low match, and the two images may be classified as belonging to different categories.

[0005] Based on the above reasoning, it can be seen that the current training method for generative models makes the hash code of the image output by the trained encoder more comprehensive and contain all the information of the corresponding image content, which may reduce the accuracy of image retrieval and classification. Summary of the Invention

[0006] The embodiments of the present application provide a method, apparatus, device, and storage medium for training an encoder, which can improve the accuracy of the encoder in classifying and retrieving images. The technical solution is as follows:

[0007] In a first aspect, a method for training an encoder is provided, the method comprising:

[0008] Acquire a first sample image and a second sample image belonging to the same image category;

[0009] Inputting the first sample image and the second sample image into an encoder respectively to obtain a first hash code corresponding to the first sample image and a second hash code corresponding to the second sample image;

[0010] determining a comparison loss between the first hash code and the second hash code;

[0011] The encoder is trained based on the contrastive loss.

[0012] Optionally, obtaining a first sample image and a second sample image belonging to the same image category includes:

[0013] A target image is acquired, and random data augmentation processing is performed twice on the target image to obtain a first sample image and a second sample image belonging to the same image category.

[0014] Optionally, the training the encoder based on the contrastive loss includes:

[0015] determining a first loss value corresponding to the first hash code and a second loss value corresponding to the second hash code based on the contrast loss and an information bottleneck optimization equation, respectively, wherein the information bottleneck optimization equation includes mutual information between the hash code corresponding to the sample image and the image label corresponding to the sample image, and mutual information between the hash code corresponding to the sample image and the sample image;

[0016] The encoder is trained based on the first loss value and the second loss value respectively.

[0017] Optionally, determining a first loss value corresponding to the first hash code and a second loss value corresponding to the second hash code based on the contrast loss and a preset information bottleneck optimization equation includes:

[0018] Determining the contrast loss as a first mutual information between the first hash code and an image label corresponding to a first sample image; performing variational estimation on the first sample image to determine a second mutual information between the first hash code and the first sample image; and determining the first loss value based on the determined first mutual information, the second mutual information, and the information bottleneck optimization equation;

[0019] The contrast loss is determined as the third mutual information between the second hash code and the image label corresponding to the second sample image; variational estimation is performed on the second sample image to determine the fourth mutual information between the second hash code and the second sample image; and the second loss value is determined based on the determined third mutual information, the fourth mutual information, and the information bottleneck optimization equation, wherein the image label corresponding to the first sample image is the same as the image label corresponding to the second sample image.

[0020] In a second aspect, a device for training an encoder is provided, the device comprising:

[0021] An acquisition module, configured to acquire a first sample image and a second sample image belonging to the same image category;

[0022] an input module, configured to input the first sample image and the second sample image into an encoder respectively, to obtain a first hash code corresponding to the first sample image and a second hash code corresponding to the second sample image;

[0023] a determining module, configured to determine a comparison loss between the first hash code and the second hash code;

[0024] A training module is used to train the encoder based on the contrast loss.

[0025] Optionally, the acquisition module is used to:

[0026] A target image is acquired, and random data augmentation processing is performed twice on the target image to obtain a first sample image and a second sample image belonging to the same image category.

[0027] Optionally, the training module is used to:

[0028] determining a first loss value corresponding to the first hash code and a second loss value corresponding to the second hash code based on the contrast loss and an information bottleneck optimization equation, respectively, wherein the information bottleneck optimization equation includes mutual information between the hash code corresponding to the sample image and the image label corresponding to the sample image, and mutual information between the hash code corresponding to the sample image and the sample image;

[0029] The encoder is trained based on the first loss value and the second loss value respectively.

[0030] Optionally, the training module is used to:

[0031] Determining the contrast loss as a first mutual information between the first hash code and an image label corresponding to a first sample image; performing variational estimation on the first sample image to determine a second mutual information between the first hash code and the first sample image; and determining the first loss value based on the determined first mutual information, the second mutual information, and the information bottleneck optimization equation;

[0032] The contrast loss is determined as the third mutual information between the second hash code and the image label corresponding to the second sample image; variational estimation is performed on the second sample image to determine the fourth mutual information between the second hash code and the second sample image; and the second loss value is determined based on the determined third mutual information, the fourth mutual information, and the information bottleneck optimization equation, wherein the image label corresponding to the first sample image is the same as the image label corresponding to the second sample image.

[0033] In a third aspect, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operations performed by the method for training an encoder as described in the first aspect.

[0034] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the operations performed by the method for training an encoder as described in the first aspect.

[0035] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0036] In this application, the encoder is trained using a contrast loss of hash codes corresponding to images of the same category, without relying on the hash codes of the decoder's output images for the encoder to converge. In other words, in this application, the encoder converges only when the hash codes of the two images output by the encoder have a high degree of match, that is, when the hash codes of the two images output contain more information related to the image category shared by both images. The hash codes of the images output by the encoder obtained through such training can include more information related to the image category, which can improve the accuracy of image classification and retrieval tasks when applied to image classification and retrieval tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a flow chart of an existing method for training and generating a model provided in an embodiment of the present application;

[0039] Figure 2 This is a flow chart of a method for training an encoder provided in an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of a method for training an encoder provided in an embodiment of the present application;

[0041] Figure 4 This is a flow chart of a method for training an encoder provided in an embodiment of the present application;

[0042] Figure 5 This is a flow chart of a method for training an encoder provided in an embodiment of the present application;

[0043] Figure 6 is a schematic diagram of a method for training an encoder provided in an embodiment of the present application;

[0044] Figure 7 1 is a schematic diagram of the structure of a device for training an encoder provided in an embodiment of the present application;

[0045] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0047] The present application provides a method for training an encoder, which relates to the technical field of artificial intelligence.

[0048] Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or digital computer-controlled machine simulation to extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. For example, in the embodiments provided in this application, image data can be processed according to artificial intelligence technology to obtain the hash code of the image, and then the matching degree of the hash codes of the two images can be used to determine whether the two images are close in display content, thereby replacing manual inspection to see whether the display content of the two images is close.

[0049] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0050] Computer vision (CV) technology is the study of how machines can "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring objects, and further processing the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0051] Key technologies in speech technology include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech becoming one of the most promising methods of human-computer interaction.

[0052] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0053] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0054] Autonomous driving technology usually includes high-precision maps, environmental perception, behavioral decision-making, path planning, motion control and other technologies. Autonomous driving technology has broad application prospects.

[0055] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0056] The solutions provided in the embodiments of this application may involve computer vision, machine learning and other technologies in artificial intelligence technology, and can be applied in image retrieval, image classification, intelligent medical care and other fields, which can be specifically illustrated through the embodiments provided in this application.

[0057] The method for training an encoder provided in this application can be implemented by a computer device.

[0058] The computer device may be a terminal or a server. When the computer device is a terminal, the terminal may be, but is not limited to, a mobile phone, tablet computer, smart wearable device, desktop computer, laptop computer, or other device with data processing capabilities. When the computer device is a server, the server may be a single server or a server group. If it is a single server, the server may be responsible for all processing in the following scheme. If it is a server group, different servers in the server group may be responsible for different processing in the following scheme. The specific processing allocation can be arbitrarily configured by technical personnel based on actual needs and will not be detailed here.

[0059] The computer device may include a memory and a processor. The memory may store data and instructions corresponding to the method for training an encoder provided herein. The data may be various sample images, and the instructions may be program code corresponding to the method for training an encoder. The processor may process the data stored in the memory according to the instructions stored in the memory to implement the method for training an encoder provided herein.

[0060] To facilitate understanding of the embodiments of the present application, some terms and concepts that may be involved in the present application are first explained:

[0061] Data augmentation is a method of processing images, including but not limited to flipping, rotating, translating, scaling, cropping, and Gaussian blurring.

[0062] Mutual information is the amount of information about another random variable contained in one random variable. In information bottleneck theory, mutual information can be used to represent the amount of information contained in two data. It can be expressed as I(X; Y), where X and Y represent two different data.

[0063] The information bottleneck theory is a method in information theory that can be used to compress data so that the compressed data retains the feature information that is most relevant to the concept label corresponding to the data as much as possible.

[0064] The generative model is a machine learning model used to implement the unsupervised image hashing method. In the prior art, when the generative model is trained, the structure of the generative model can be as follows: Figure 1 As shown, it includes an encoder and a decoder. Figure 1 Where X is the image data input to the encoder, and X' is the image data restored by the decoder. In the method for training an encoder provided in the present application, the encoder can be trained directly without the aid of a decoder.

[0065] Figure 2 This is a flow chart of a method for training an encoder provided by an embodiment of the present application. Figure 2 , the embodiment includes:

[0066] Step 201: Acquire a first sample image and a second sample image belonging to the same image category.

[0067] Among them, the category of the image can be set by the technician according to the downstream task corresponding to the trained encoder. For example, if the corresponding downstream task is a medical image retrieval task, the category of the image can be distinguished according to the symptoms corresponding to the pathological site displayed in the medical image, that is, medical images showing the same pathological site and corresponding to the same symptoms can be considered as medical images of the same image category. Among them, the medical image can be an image of the patient's body taken directly by a camera (such as an image of the skin of a dermatitis patient), or an image of the patient's body taken by an endoscope, or an X-ray film taken by an X-ray machine, etc. The technician can obtain sample images corresponding to different image categories by obtaining historical medical record information, and then classify the obtained sample images to obtain a first sample image and a second sample image belonging to the same image category.

[0068] Optionally, the present application also provides a method for obtaining sample images, which does not require technical personnel to classify the sample images. The corresponding processing is as follows: obtain a target image, perform two random data enhancement processes on the target image, and obtain a first sample image and a second sample image belonging to the same image category.

[0069] In practice, technicians can directly obtain a target image, where the target image can be any sample image corresponding to the encoder's downstream task. Each target image is then subjected to two random data augmentation processes. After the two random data augmentation processes, two processed sample images can be obtained. These two sample images are referred to as the first sample image and the second sample image. The random data augmentation process can include flipping the sample image based on a random flip angle, cropping the flipped sample image based on a random cropping size, and then Gaussian blurring the cropped sample image based on a random radius. After the target image is subjected to the two random data augmentation processes, the first and second sample images may have some differences in displayed content, but they still belong to the same category. For example, after a target image containing the character "cat" is subjected to the two random data augmentation processes, the first and second sample images may have more or less differences in size, color, clarity, etc. However, the first and second sample images still contain information about the character "cat" and belong to the same category of images.

[0070] It can be seen that by adopting the present application, technicians do not need to pre-classify the sample images, and can obtain sample images belonging to the same image category through random data enhancement processing, which can improve the efficiency of obtaining sample images.

[0071] Step 202: Input the first sample image and the second sample image into an encoder respectively to obtain a first hash code corresponding to the first sample image and a second hash code corresponding to the second sample image.

[0072] In this application, when the encoder is trained, the decoder does not need to participate in the training process. In implementation, after obtaining the first sample image and the second sample image, the first sample image and the second sample image can be input into the encoder during the training process. The encoder outputs the first hash code corresponding to the first sample image and the second hash code corresponding to the second sample image, and the processing is as follows:

[0073] Given a set of target images x (k) , where k = 1, 2, ... N represents the number of the target image. For each target image x, two random data enhancement processes can be performed to obtain the first sample image corresponding to the target image And the second sample image Then you can Input to encoder f θ (v) as follows:

[0074]

[0075] In order to obtain the hash codes corresponding to the first sample image and the second sample image, a probability model can be further introduced as follows:

[0076]

[0077] Where σ represents a sigmoid function (an S-shaped function). The hash codes corresponding to the first sample image and the second sample image can be obtained by sampling the multidimensional Bernoulli distribution as follows:

[0078]

[0079] In addition, the process of obtaining the hash code corresponding to the first sample image and the second sample image involves the sampling process, so the obtained hash code is not differentiable. In order to use the gradient descent algorithm to minimize the contrast loss when training the encoder later, the gradient estimator (straight through) can also be used to perform gradient estimation, as follows:

[0080]

[0081] Here, u is a preset parameter, representing a uniform distribution between [0, 1]. Its specific value can be preset by a technician, for example, it can be set to 0.4, 0.5, 0.6, etc.

[0082] Step 203: Determine the comparison loss between the first hash code and the second hash code.

[0083] like Figure 3 As shown, through the target image x, the first sample image is obtained and the second sample image After that, the first sample image and the second sample image can be input into the encoder in the training process respectively. The encoder outputs the first hash code corresponding to the first sample image And the second hash code corresponding to the second sample image Then you can use Determine the corresponding contrast loss L.

[0084] because They are obtained from the first sample image and the second sample image after two data augmentation processes corresponding to the same target image, so they should contain similar semantic information. Among them, the contrast loss can be obtained by a preset contrast loss function.

[0085] Step 204: Train the encoder based on the contrast loss.

[0086] After obtaining the contrast loss between the first hash code and the second hash code, the encoder can be trained using a gradient descent method. The process of training the encoder using the gradient descent method is conventional and will not be described in detail here.

[0087] After extensive training of the encoder, or when the encoder parameters converge to a certain level, the encoder training can be considered complete. After obtaining the trained encoder, the encoder can be applied to downstream tasks such as image retrieval and image classification.

[0088] In this application, the encoder is trained using a contrast loss of hash codes corresponding to images of the same category, without relying on the hash codes of the decoder's output images for the encoder to converge. In other words, in this application, the encoder converges only when the hash codes of the two images output by the encoder have a high degree of match, that is, when the hash codes of the two images output contain more information related to the image category shared by both images. The hash codes of the images output by the encoder obtained through such training can include more information related to the image category, which can improve the accuracy of image classification and retrieval tasks when applied to image classification and retrieval tasks.

[0089] Figure 4 This is a flow chart of a method for training an encoder provided by an embodiment of the present application. Figure 4 , the embodiment includes:

[0090] Step 401: Acquire a first sample image and a second sample image belonging to the same image category.

[0091] Step 402: Input the first sample image and the second sample image into an encoder respectively to obtain a first hash code corresponding to the first sample image and a second hash code corresponding to the second sample image.

[0092] Step 403: Determine the comparison loss between the first hash code and the second hash code.

[0093] The processing of the above steps 401-403 is the same as the processing of the above steps 201-203, and will not be repeated here.

[0094] Step 404 : Based on the contrast loss and a preset information bottleneck optimization equation, a first loss value corresponding to the first hash code and a second loss value corresponding to the second hash code are determined respectively.

[0095] The information bottleneck optimization equation includes two terms. The first term in the information bottleneck optimization equation is the mutual information between the hash code corresponding to the sample image and the image label corresponding to the sample image. The image label corresponding to the sample image can be used to identify the category of the sample image. Sample images of the same category have the same image label. The second term in the information bottleneck optimization equation is the mutual information between the hash code corresponding to the sample image and the sample image, as follows:

[0096]

[0097] Among them, max θ R IB is the optimization target, θ represents the parameters in the encoder, I(Y,B) represents the mutual information between the hash code corresponding to the sample image and the image label corresponding to the sample image. Although I(Y,B) is difficult to calculate directly, if the mutual information between the hash code corresponding to the sample image and the image label is larger, the hash code corresponding to the sample image can more reflect the information of the image label corresponding to the sample image. When the hash code corresponding to the sample image can more reflect the image label information of the sample image, the first hash code corresponding to the first sample image with the same image label and the second hash code corresponding to the second sample image are closer, that is, the contrast loss between the first hash code and the second hash code is smaller. Therefore, the optimization of the mutual information between the hash code corresponding to the sample image and the image label can be transformed into the optimization of the contrast loss between the first hash code and the second hash code. Therefore, the corresponding optimization equation is as follows:

[0098]

[0099] Where β is a preset parameter, and I(B,V) is used to represent the mutual information between the hash code corresponding to the sample image and the sample image, which can be determined by performing variational estimation on the sample image as follows:

[0100] I(B,V)≤E p(v) [KL(p(b|v)||q(b))]

[0101] Among them, E p(v) represents the expectation, and the calculation formula corresponding to KL(p(b|v)||q(b)) is as follows:

[0102]

[0103] Wherein γ is a parameter of the variational distribution, and its specific value can be preset by a technician, for example, it can be 0.5, D represents the dimension of the hash code, and d represents each dimension of the hash code.

[0104] As can be seen above, the information bottleneck optimization equation can be converted into the following equation:

[0105]

[0106] Based on the above reasoning, the corresponding process of determining the first loss value corresponding to the first hash code and the second loss value corresponding to the second hash code is as follows:

[0107] (1) Determine the contrast loss as the first mutual information between the first hash code and the image label corresponding to the first sample image; perform variational estimation on the first sample image to determine the second mutual information between the first hash code and the first sample image, and determine the first loss value based on the determined first mutual information, the second mutual information, and the information bottleneck optimization equation.

[0108] In implementation, the contrast loss corresponding to the first hash code and the second hash code can be used to determine the mutual information corresponding to the first term in the information bottleneck optimization equation (i.e., the first mutual information), and then variational estimation is performed on the first sample image to obtain the second mutual information between the first hash code and the first sample image.

[0109] Among them, variational estimation belongs to the existing technology and will not be described in detail here. It can be based on the above formula I(B,V)≤E p(v) [KL(p(b|v)||q(b))] is implemented. After obtaining the first mutual information and the second mutual information, the corresponding first loss value can be determined based on the above information bottleneck optimization equation. That is, the first loss value is equal to the difference between the first mutual information and β times the second mutual information.

[0110] (2) Determine the contrast loss as the third mutual information between the second hash code and the image label corresponding to the second sample image; perform variational estimation on the second sample image to determine the fourth mutual information between the second hash code and the second sample image, and determine the second loss value based on the determined third mutual information, fourth mutual information, and information bottleneck optimization equation, wherein the image label corresponding to the first sample image is the same as the image label corresponding to the second sample image.

[0111] In implementation, the contrast loss corresponding to the first hash code and the second hash code can be used to determine the mutual information corresponding to the first term in the information bottleneck optimization equation (i.e., the third mutual information), and then variational estimation is performed on the second sample image to obtain the fourth mutual information between the second hash code and the second sample image.

[0112] Among them, variational estimation belongs to the prior art and will not be described in detail here. It can be based on the above formula I(B,V)≤E p(v) [KL(p(b|v)||q(b))] is implemented. After obtaining the third and fourth mutual information, the corresponding second loss value can be determined based on the above information bottleneck optimization equation. That is, the second loss value is equal to the difference between the third mutual information and β times the fourth mutual information.

[0113] like Figure 5 As shown, in the embodiment of the present application, the target image x can be subjected to two random data enhancement processes to obtain a first sample image and the second sample image After that, the first sample image and the second sample image can be input into the encoder in the training process respectively. The encoder outputs the first hash code corresponding to the first sample image And the second hash code corresponding to the second sample image Then you can use Determine the first sample image in the information bottleneck optimization equation respectively and the second sample image Mutual information I(Y,B) with the corresponding image label, where Y is the first sample image and the second sample image The corresponding image label. And can be based on the first sample image and the second sample image And the corresponding first hash code and the second hash code Determine the first sample image respectively and the second sample image And the mutual information I(B,V) of the corresponding hash code, (that is, the first sample image and the second sample image Then, the first hash code can be calculated based on the mutual information and information bottleneck optimization equations obtained above. The corresponding first loss value and second hash code The corresponding second loss value.

[0114] Step 405: Train the encoder based on the first loss value and the second loss value respectively.

[0115] In implementation, after obtaining the first and second loss values, the encoder can be trained based on the first and second loss values, respectively. Specifically, the encoder is trained once using the first loss value and then again using the second loss value. Training the encoder using the loss values ​​can be accomplished using a gradient descent method, which is a prior art technique and will not be described in detail here.

[0116] After extensive training of the encoder, or when the encoder parameters converge to a certain level, the encoder training can be considered complete. After obtaining the trained encoder, the encoder can be applied to downstream tasks such as image retrieval and image classification.

[0117] For example, the downstream task is a medical image retrieval task. The technician can set the trained encoder in the medical image retrieval program, wherein the medical image retrieval program can be a processing program set in a low-resource device such as a terminal, or a background program set in a server with stronger processing power. After the terminal user obtains the medical image, he can upload the corresponding medical image to the corresponding interface of the medical image retrieval program, such as Figure 6 As shown, the subsequent terminal or server can input the uploaded medical image into the encoder to obtain the hash code of the medical image, and then determine at least one medical image that matches the hash code in the corresponding medical image database, and then display the matched medical image in the Figure 6 In the interface shown, the medical image database may include historical medical record information, that is, various medical images, symptoms corresponding to each medical image, and treatment methods for the symptoms corresponding to each medical image.

[0118] In this application, the encoder is trained using a contrast loss of hash codes corresponding to images of the same category, without relying on the hash codes of the decoder's output images for the encoder to converge. In other words, in this application, the encoder converges only when the hash codes of the two images output by the encoder have a high degree of match, that is, when the hash codes of the two images output contain more information related to the image category shared by both images. The hash codes of the images output by the encoder obtained through such training can include more information related to the image category, which can improve the accuracy of image classification and retrieval tasks when applied to image classification and retrieval tasks.

[0119] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0120] The present application also provides a device for training an encoder, which may be the terminal or server of the above embodiment, such as Figure 7 As shown, the device includes:

[0121] An acquisition module 710 is configured to acquire a first sample image and a second sample image belonging to the same image category;

[0122] An input module 720 is configured to input the first sample image and the second sample image into an encoder respectively to obtain a first hash code corresponding to the first sample image and a second hash code corresponding to the second sample image;

[0123] a determination module 730, configured to determine a comparison loss between the first hash code and the second hash code;

[0124] The training module 740 is configured to train the encoder based on the contrastive loss.

[0125] Optionally, the acquisition module 710 is configured to:

[0126] A target image is acquired, and random data augmentation processing is performed twice on the target image to obtain a first sample image and a second sample image belonging to the same image category.

[0127] Optionally, the training module 740 is used to:

[0128] determining a first loss value corresponding to the first hash code and a second loss value corresponding to the second hash code based on the contrast loss and an information bottleneck optimization equation, respectively, wherein the information bottleneck optimization equation includes mutual information between the hash code corresponding to the sample image and the image label corresponding to the sample image, and mutual information between the hash code corresponding to the sample image and the sample image;

[0129] The encoder is trained based on the first loss value and the second loss value respectively.

[0130] Optionally, the training module 740 is used to:

[0131] Determining the contrast loss as a first mutual information between the first hash code and an image label corresponding to a first sample image; performing variational estimation on the first sample image to determine a second mutual information between the first hash code and the first sample image; and determining the first loss value based on the determined first mutual information, the second mutual information, and the information bottleneck optimization equation;

[0132] The contrast loss is determined as the third mutual information between the second hash code and the image label corresponding to the second sample image; variational estimation is performed on the second sample image to determine the fourth mutual information between the second hash code and the second sample image; and the second loss value is determined based on the determined third mutual information, the fourth mutual information, and the information bottleneck optimization equation, wherein the image label corresponding to the first sample image is the same as the image label corresponding to the second sample image.

[0133] It should be noted that the apparatus for training an encoder provided in the above embodiment only uses the division of the above functional modules as an example for illustrating the training of the encoder. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for training an encoder provided in the above embodiment and the method embodiment for training an encoder are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0134] Figure 8 The following is a block diagram of a computer device 800 according to an exemplary embodiment of the present application. The computer device 800 may be a terminal or server as described in the aforementioned embodiments. The computer device 800 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computer device 800 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other similar terminology.

[0135] Typically, the computer device 800 includes a processor 801 and a memory 802 .

[0136] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (digital signal processing), FPGA (field-programmable gate array), or PLA (programmable logic array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (central processing unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (graphics processing unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (artificial intelligence) processor, which is used to process computing operations related to machine learning.

[0137] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, which is used to be executed by the processor 801 to implement the method for training the encoder provided in the method embodiment of the present application.

[0138] In some embodiments, computer device 800 may optionally include a peripheral device interface 803 and at least one peripheral device. Processor 801, memory 802, and peripheral device interface 803 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 803 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, and a power supply 809.

[0139] The peripheral device interface 803 can be used to connect at least one I / O (input / output)-related peripheral device to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0140] The radio frequency circuit 804 is used to receive and transmit RF (radio frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 804 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (wireless fidelity) network. In some embodiments, the radio frequency circuit 804 may also include circuits related to NFC (near field communication), which is not limited in this application.

[0141] Display screen 805 is used to display a user interface (UI). This UI can include graphics, text, icons, videos, or any combination thereof. If display screen 805 is a touchscreen display, it can also capture touch signals on or above the surface of display screen 805. These touch signals can be input as control signals to processor 801 for processing. Display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 805, located on the front panel of computer device 800. In other embodiments, there can be at least two display screens 805, located on different surfaces of computer device 800 or in a foldable design. In still other embodiments, display screen 805 can be a flexible display screen, located on a curved or foldable surface of computer device 800. Display screen 805 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 805 can be made of materials such as LCD (liquid crystal display) and OLED (organic light-emitting diode).

[0142] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (virtual reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0143] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 801 for processing, or input into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, each located in different parts of the computer device 800. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as distance measurement. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0144] Power supply 809 is used to power various components in computer device 800. Power supply 809 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 809 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0145] In some embodiments, the computer device 800 further includes one or more sensors 810 , including but not limited to: an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , an optical sensor 815 , and a proximity sensor 816 .

[0146] The accelerometer 811 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the computer device 800. For example, the accelerometer 811 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 801 can control the display screen 805 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 811. The accelerometer 811 can also be used to collect game or user motion data.

[0147] The gyroscope sensor 812 can detect the orientation and rotation angle of the computer device 800. It can also work with the accelerometer 811 to collect 3D motions of the user on the computer device 800. Based on the data collected by the gyroscope sensor 812, the processor 801 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0148] The pressure sensor 813 can be installed on the side frame of the computer device 800 and / or below the display screen 805. When the pressure sensor 813 is installed on the side frame of the computer device 800, it can detect the user's grip signal of the computer device 800. The processor 801 can perform left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 813. When the pressure sensor 813 is installed below the display screen 805, the processor 801 controls the operational controls on the UI interface based on the user's pressure operation on the display screen 805. The operational controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0149] The optical sensor 815 is used to detect ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 based on the ambient light intensity detected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is decreased. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 based on the ambient light intensity detected by the optical sensor 815.

[0150] Proximity sensor 816, also known as a distance sensor, is typically located on the front panel of computer device 800. Proximity sensor 816 is used to detect the distance between the user and the front of computer device 800. In one embodiment, when proximity sensor 816 detects that the distance between the user and the front of computer device 800 is gradually decreasing, processor 801 controls display screen 805 to switch from the screen-on state to the screen-off state. When proximity sensor 816 detects that the distance between the user and the front of computer device 800 is gradually increasing, processor 801 controls display screen 805 to switch from the screen-off state to the screen-on state.

[0151] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the computer device 800, and the computer device 800 may include more or fewer components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0152] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to perform the method for training an encoder in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (read-only memory), a RAM (random access memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0153] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0154] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for training an encoder, characterized in that: The method comprises: Acquire a first sample image and a second sample image belonging to the same image category; Inputting the first sample image and the second sample image into an encoder respectively to obtain a first hash code corresponding to the first sample image and a second hash code corresponding to the second sample image; determining a comparison loss between the first hash code and the second hash code; determining a first loss value corresponding to the first hash code and a second loss value corresponding to the second hash code based on the contrast loss and an information bottleneck optimization equation, respectively, wherein the information bottleneck optimization equation includes mutual information between the hash code corresponding to the sample image and the image label corresponding to the sample image, and mutual information between the hash code corresponding to the sample image and the sample image; The encoder is trained based on a gradient descent method and the first loss value, and the encoder is trained based on a gradient descent method and the second loss value.

2. The method according to claim 1, characterized in that The acquiring of the first sample image and the second sample image belonging to the same image category includes: A target image is acquired, and random data augmentation processing is performed twice on the target image to obtain a first sample image and a second sample image belonging to the same image category.

3. The method according to claim 1, characterized in that The determining, based on the contrast loss and the information bottleneck optimization equation, a first loss value corresponding to the first hash code and a second loss value corresponding to the second hash code, respectively, includes: Determining the contrast loss as a first mutual information between the first hash code and an image label corresponding to a first sample image; performing variational estimation on the first sample image to determine a second mutual information between the first hash code and the first sample image; and determining the first loss value based on the determined first mutual information, the second mutual information, and the information bottleneck optimization equation; The contrast loss is determined as the third mutual information between the second hash code and the image label corresponding to the second sample image; variational estimation is performed on the second sample image to determine the fourth mutual information between the second hash code and the second sample image; and the second loss value is determined based on the determined third mutual information, the fourth mutual information, and the information bottleneck optimization equation, wherein the image label corresponding to the first sample image is the same as the image label corresponding to the second sample image.

4. A device for training an encoder, characterized in that The device comprises: An acquisition module, configured to acquire a first sample image and a second sample image belonging to the same image category; an input module, configured to input the first sample image and the second sample image into an encoder respectively, to obtain a first hash code corresponding to the first sample image and a second hash code corresponding to the second sample image; a determining module, configured to determine a comparison loss between the first hash code and the second hash code; A training module is configured to determine, based on the contrast loss and an information bottleneck optimization equation, a first loss value corresponding to the first hash code and a second loss value corresponding to the second hash code, respectively, wherein the information bottleneck optimization equation includes the mutual information between the hash code corresponding to the sample image and the image label corresponding to the sample image, and the mutual information between the hash code corresponding to the sample image and the sample image; train the encoder based on a gradient descent method and the first loss value, and train the encoder based on a gradient descent method and the second loss value.

5. The device according to claim 4, characterized in that The acquisition module is used to: A target image is acquired, and random data augmentation processing is performed twice on the target image to obtain a first sample image and a second sample image belonging to the same image category.

6. The device according to claim 4, characterized in that The training module is used to: Determining the contrast loss as a first mutual information between the first hash code and an image label corresponding to a first sample image; performing variational estimation on the first sample image to determine a second mutual information between the first hash code and the first sample image; and determining the first loss value based on the determined first mutual information, the second mutual information, and the information bottleneck optimization equation; Determining the contrast loss as a third mutual information between the second hash code and an image label corresponding to the second sample image; performing variational estimation on the second sample image to determine a fourth mutual information between the second hash code and the second sample image, and determining the second loss value based on the determined third mutual information, the fourth mutual information, and the information bottleneck optimization equation, wherein the image label corresponding to the first sample image is the same as the image label corresponding to the second sample image.

7. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operation performed by the method for training an encoder according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operation performed by the method for training an encoder according to any one of claims 1 to 3.

Images