Model training method, image processing method, device, medium and electronic equipment

Abstract

The present disclosure relates to a model training method, an image processing method, an apparatus, a medium and an electronic device. The model training method comprises: obtaining an original image and an enhanced image obtained by performing image enhancement on the original image; inputting the enhanced image into at least two auxiliary encoders respectively to obtain first image features corresponding to the outputs of the auxiliary encoders, and inputting the original image into a main encoder to obtain second image features output by the main encoder; performing contrast learning on the main encoder according to the first image features, the second image features and negative samples obtained, to obtain a trained main encoder. In the contrast learning process, the parameters of the main encoder are updated using a back propagation algorithm, and the parameters of the auxiliary encoders are updated using a momentum update algorithm. The momentum coefficients used in the momentum update algorithms corresponding to the auxiliary encoders are different, which improves the feature expression capability of the trained main encoder for target images.

Description

Technical Field

[0001] This disclosure relates to the field of electronic information technology, and more specifically, to a model training method, an image processing method, an apparatus, a medium, and an electronic device. Background Technology

[0002] In feature network learning, the most common application of self-supervised learning is to pre-train the feature network with unlabeled data and then transfer the learned knowledge to downstream tasks. The better the pre-training effect of self-supervised learning, the better the performance on downstream tasks.

[0003] Therefore, improving the training effect of feature network pre-training is crucial. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Firstly, this disclosure provides a model training method, including:

[0006] Acquire the original image and the enhanced image obtained by performing image enhancement on the original image;

[0007] The enhanced image is input into at least two auxiliary encoders to obtain the first image features output by each auxiliary encoder, and the original image is input into the main encoder to obtain the second image features output by the main encoder.

[0008] The master encoder is trained by comparative learning based on the first image features, the second image features, and the acquired negative samples. The trained master encoder is used to extract image features of the target image. During the comparative learning process, the parameters of the master encoder are updated by backpropagation algorithm, and the parameters of each of the auxiliary encoders are updated by momentum update algorithm. The momentum coefficients used in the momentum update algorithm for each of the auxiliary encoders are different.

[0009] Secondly, this disclosure provides an image processing method, including:

[0010] Acquire the target image;

[0011] The target image is processed according to the image processing model to obtain the image processing result, wherein the image processing model is obtained by training the decoder and the master encoder trained in the first aspect.

[0012] Thirdly, this disclosure provides a model training apparatus, comprising:

[0013] The first acquisition module is used to acquire the original image and the enhanced image obtained by image enhancement of the original image;

[0014] The output module is used to input the enhanced image into at least two auxiliary encoders respectively to obtain the first image features output by each auxiliary encoder, and to input the original image into the main encoder to obtain the second image features output by the main encoder.

[0015] The learning module is used to perform comparative learning on the main encoder based on the first image features, the second image features, and the acquired negative samples to obtain a trained main encoder. The trained main encoder is used to extract image features of the target image. During the comparative learning process, the backpropagation algorithm is used to update the parameters of the main encoder, and the momentum update algorithm is used to update the parameters of each of the auxiliary encoders. The momentum coefficients used in the momentum update algorithm for each of the auxiliary encoders are different.

[0016] Fourthly, this disclosure provides an image processing apparatus, comprising:

[0017] The second acquisition module is used to acquire the target image;

[0018] The processing module is used to process the target image according to the image processing model to obtain the image processing result, wherein the image processing model is obtained by training the decoder and the master encoder trained in the first aspect.

[0019] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect or the steps of the method described in the second aspect.

[0020] Sixthly, this disclosure provides an electronic device, comprising:

[0021] A storage device on which computer programs are stored;

[0022] A processing device for executing the computer program in the storage device to implement the steps of the method in the first aspect or the steps of the method in the second aspect.

[0023] Through the above technical solution, in the comparative learning process, the momentum coefficients used in the momentum update algorithms corresponding to each co-encoder are different. The magnitude of the momentum coefficient determines the update speed of the co-encoder's parameters. Compared with the co-encoder using a larger momentum coefficient, the former updates its parameters faster, and its update speed is closer to that of the main encoder, thus capturing the detailed features of the image. The latter updates its parameters slower, thus capturing the structural features of the image. By using these two aspects of detailed features and structural features to assist the main encoder in comparative learning, the feature representation ability of the trained main encoder of the target image can be improved. In turn, when the trained main encoder is applied to the image processing model corresponding to the downstream task, the performance of the image processing model can be improved.

[0024] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0025] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0026] Figure 1 This is a flowchart illustrating a model training method according to an exemplary embodiment.

[0027] Figure 2 This is a framework diagram illustrating a contrastive learning dependency according to an exemplary embodiment.

[0028] Figure 3 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0029] Figure 4 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment.

[0030] Figure 5 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0031] Figure 6 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0039] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0040] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0041] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0042] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0043] In feature network learning, the most common application of self-supervised learning is to pre-train the feature network on unlabeled data and then transfer the learned knowledge to downstream tasks. The better the pre-training effect of self-supervised learning, the better the performance on the downstream task. Therefore, improving the training effect of feature network pre-training is crucial. This is especially true for training feature networks based on long-tail sample sets. In a long-tail sample set, the classes with fewer samples are called tail classes, and the classes with more samples are called head classes. Due to the large differences in this data distribution, models trained on such sample sets are prone to decision bias towards head classes, leading to a decline in model performance.

[0044] In view of this, embodiments of the present disclosure provide a model training method, an image processing method, an apparatus, a medium, and an electronic device to improve the feature representation ability of the trained master encoder for the target image, thereby improving the performance of the image processing model when the trained master encoder is applied to the image processing model corresponding to the downstream task.

[0045] The embodiments of this disclosure will be further explained below with reference to the accompanying drawings.

[0046] Figure 1 This is a flowchart illustrating a model training method according to an exemplary embodiment, with reference to... Figure 1 The model training method may include the following steps:

[0047] Step 101: Obtain the original image and the enhanced image obtained by performing image enhancement on the original image.

[0048] The original image can be a medical image.

[0049] Image enhancement involves making a series of random changes to the original image to generate similar but different training samples, thereby expanding the size of the training dataset. Furthermore, randomly changing the training samples can reduce the model's dependence on certain attributes, thus improving the model's generalization ability.

[0050] For example, image enhancement can include one or more combined data enhancement transformations such as multiple cropping, flipping, random affine transformation, reflection, scaling transformation, translation transformation, scale transformation, contrast transformation, noise perturbation, and color transformation to increase the size of the dataset, and flipping can be up, down, left, or right.

[0051] Step 102: Input the enhanced image into at least two auxiliary encoders to obtain the first image features output by each auxiliary encoder, and input the original image into the main encoder to obtain the second image features output by the main encoder.

[0052] Both the auxiliary encoder and the main encoder are used to extract features from the input image. These features can be represented by vectors, similar to the function of the aforementioned feature network. The network structures of the auxiliary encoder and the main encoder are similar. The following explanation uses the main encoder as an example.

[0053] The main encoder may include a first network and a second network, which are connected. The original image passes through the first network to obtain high-dimensional image features, and the second network performs dimensionality reduction processing on the high-dimensional image features to obtain second image features. For example, the first network may be a ResNet-50 network, and the second network may be a two-layer linear layer, where the output of the last layer in the linear layer can be the second image features.

[0054] Step 103: Based on the first image features, the second image features, and the obtained negative samples, the main encoder is subjected to comparative learning to obtain the trained main encoder. The trained main encoder is used to extract image features of the target image. During the comparative learning process, the backpropagation algorithm is used to update the parameters of the main encoder, and the momentum update algorithm is used to update the parameters of each auxiliary encoder. The momentum coefficient used in the momentum update algorithm corresponding to each auxiliary encoder is different.

[0055] It is worth noting that contrastive learning is a self-supervised learning method used to learn general features of the original images in a sample set by having the main encoder learn which data points are similar or different without labels.

[0056] The negative sample queue maintains negative samples, while the original image is used as a positive sample.

[0057] The process of comparative learning, the parameters of the main encoder, and the updating of the parameters of the auxiliary encoder can be referred to in the following relevant embodiments, which will not be elaborated here.

[0058] In the comparative learning process described above, the momentum coefficients used in the momentum update algorithms of each co-encoder are different. The magnitude of the momentum coefficient determines the update speed of the co-encoder's parameters. Compared with a co-encoder using a larger momentum coefficient, the former updates its parameters faster, making its update speed closer to that of the main encoder, thus capturing the detailed features of the image. The latter updates its parameters more slowly, thus capturing the structural features of the image. By using both detailed and structural features to assist the main encoder in comparative learning, the trained main encoder's ability to represent the features of the target image can be improved. Consequently, when the trained main encoder is applied to the image processing model corresponding to the downstream task, the performance of the image processing model can be improved.

[0059] In some embodiments, the step of performing comparative learning on the main encoder based on each first image feature, second image feature, and acquired negative samples to obtain a trained main encoder may include: for each auxiliary encoder, determining the loss value corresponding to the auxiliary encoder based on the negative samples in the acquired negative sample queue corresponding to the auxiliary encoder, the first image feature corresponding to the auxiliary encoder, and the second image feature; updating the parameters of the main encoder using a backpropagation algorithm based on all loss values, and updating the parameters of each auxiliary encoder using a momentum update algorithm corresponding to each auxiliary encoder based on the updated parameters of the main encoder, until a first convergence condition is met, thereby obtaining a trained main encoder.

[0060] The negative samples maintained in the negative sample queue corresponding to each co-encoder can be different, but the number of negative samples maintained can be the same.

[0061] The loss value corresponding to the co-encoder can be determined by the following formula (1) based on the negative samples in the negative sample queue corresponding to the co-encoder, the first image feature corresponding to the co-encoder, and the second image feature:

[0062]

[0063] Among them, L i Let p be the loss value corresponding to the i-th co-encoder, p be the second image feature, and ω be the loss value. i ω represents the first image feature output by the i-th co-encoder. n For the nth negative sample in the negative sample queue corresponding to the i-th co-encoder, τ iThe preset temperature coefficient is the one corresponding to the i-th auxiliary encoder. N is the number of negative samples in the negative sample queue corresponding to the i-th auxiliary encoder. exp() is an exponential function with the natural constant e as the base. sim() is a similarity function. sim() is used to calculate the distance between two vectors, which is used to characterize the degree of similarity between the two vectors. log() is a logarithmic function.

[0064] Here, backpropagation refers to gradient descent, which means modifying network parameters according to the value of a preset loss function. The preset loss function is represented by the above equation (1). As can be seen from the above equation (1), only the distance between positive sample pairs is calculated in the numerator, and negative samples only appear in the denominator. When the distance between positive sample pairs is smaller, the distance between positive and negative sample pairs is larger, and the corresponding loss value is smaller. Therefore, the above equation (1) is used to calculate the loss value corresponding to each auxiliary encoder, and the parameters of the main encoder are updated with the goal of minimizing the loss value corresponding to all auxiliary encoders.

[0065] The steps described above, namely updating the parameters of the master encoder using the backpropagation algorithm based on all loss values, and updating the parameters of each auxiliary encoder using the momentum update algorithm corresponding to each auxiliary encoder based on the updated parameters of the master encoder, until the first convergence condition is met to obtain the trained master encoder, may include: weighting all loss values ​​according to preset weights to obtain a total loss value; updating the parameters of the master encoder using the backpropagation algorithm based on the total loss value, and updating the parameters of each auxiliary encoder using the momentum update algorithm corresponding to each auxiliary encoder based on the updated parameters of the master encoder, until the first convergence condition is met to obtain the trained master encoder.

[0066] Each co-encoder corresponds to a preset weight, and all loss values ​​are weighted according to the weight corresponding to each co-encoder.

[0067] The total loss value can be determined using the following formula (2):

[0068]

[0069] Where L is the total loss value, L i Let β be the loss value corresponding to the i-th co-encoder. i The preset weight is the weight corresponding to the i-th auxiliary encoder, and J is the number of auxiliary encoders. Preferably, J can be 2.

[0070] The parameters of each auxiliary encoder can be updated using the following formula, based on the updated parameters of the main encoder and the momentum update algorithm corresponding to each auxiliary encoder:

[0071] θ i ←αi θ i +(1-α i )θ p (3)

[0072] Among them, θ to the left of the arrow in equation (3) above i For the updated parameters of the i-th auxiliary encoder, θ on the right side of the arrow in equation (3) above. i θ represents the parameters of the i-th co-encoder before the update. p For the updated parameters of the master encoder, α i Let α be the momentum coefficient corresponding to the i-th auxiliary encoder, where α i ∈[0,1).

[0073] From equation (3) above, it can be seen that in α i As α increases and approaches 1, the updated parameters of the i-th co-encoder are closer to the unupdated parameters, meaning the parameters of the i-th co-encoder update more slowly; in α... i The smaller the value, or even 0, the greater the difference between the updated parameters of the i-th auxiliary encoder and the unupdated parameters. Furthermore, the updated parameters of the i-th auxiliary encoder are similar to the updated parameters of the main encoder, meaning that the parameters of the i-th auxiliary encoder are updated faster.

[0074] The first convergence condition can be that the total loss value is less than or equal to a first preset threshold, or the first convergence condition can be that the number of times the parameters of the main encoder and the auxiliary encoder are updated reaches a second preset threshold.

[0075] The following explanation uses the term "secondary encoder" to describe the present disclosure.

[0076] Figure 2 This is a framework diagram illustrating a contrastive learning dependency according to an exemplary embodiment. Figure 2 The system includes a main encoder and two auxiliary encoders, designated as Auxiliary Encoder 1 and Auxiliary Encoder 2. Auxiliary Encoder 1 includes networks A1 and A2, and Auxiliary Encoder 2 includes networks B1 and B2. The main encoder includes networks C1 and C2. Networks A1, B1, and C1 function identically to the first network described above, while networks A2, B2, and C2 function identically to the second network described above. For detailed explanations, please refer to the relevant embodiments described above. Auxiliary Encoder 1 corresponds to a negative sample queue 1, and Auxiliary Encoder 2 corresponds to a negative sample queue 2.

[0077] exist Figure 2In this process, the original image is input into network C1, which outputs a high-dimensional feature. This high-dimensional feature is then processed by network C2 for dimensionality reduction, resulting in a second image feature. The original image undergoes a first image enhancement process, resulting in an enhanced image input into network A1. Network A1 processes this enhanced image to obtain its own high-dimensional feature, which is then processed by network A2 for dimensionality reduction, resulting in a first image feature corresponding to the output of co-encoder 1. The original image undergoes a second image enhancement process, resulting in an enhanced image input into network B1. Network B1 processes this enhanced image to obtain its own high-dimensional feature, which is then processed by network B2 for dimensionality reduction, resulting in a first image feature corresponding to the output of co-encoder 2. The original image and the enhanced image can be 224x224 pixels in size, and the first and second image enhancements can be different or the same.

[0078] For the co-encoder 1, the first image feature output by it is similar to the image feature of each negative sample in the negative sample queue 1 to obtain the corresponding first similarity distance. The first similarity distance is used to represent the degree of similarity between positive and negative sample pairs. In addition, the first image feature output by the co-encoder 1 is similar to the second image feature output by the main encoder to obtain the corresponding second similarity distance. The similarity distance is used to characterize the degree of similarity between positive sample pairs.

[0079] For the co-encoder 2, the first image feature output by it is similar to the image feature of each negative sample in the negative sample queue 2 to obtain the corresponding fourth similarity distance. This fourth similarity distance is used to characterize the similarity between positive and negative sample pairs. In addition, the first image feature output by the co-encoder 2 is similar to the second image feature output by the main encoder to obtain the corresponding third similarity distance. This similarity distance is used to characterize the similarity between positive sample pairs.

[0080] By calculating the first and second similarity distances corresponding to all negative samples in negative sample queue 1, and other parameters in equation (1) above, the loss value corresponding to co-encoder 1 is determined. The loss value corresponding to co-encoder 1 is characterized by the following equation (4):

[0081]

[0082] Where L1 is the loss value corresponding to co-encoder 1, p is the second image feature, ω1 is the first image feature output by co-encoder 1, and ω k For the image features corresponding to the k-th negative sample in negative sample queue 1, sim(p,ω1) represents the second similarity distance, and sim(ω1,ω k) represents the first similarity distance corresponding to the k-th negative sample in negative sample queue 1, τ1 represents the temperature coefficient corresponding to co-encoder 1, τ1 can be 0.9, and K is the number of negative samples in negative sample queue 1, K can be 65536.

[0083] The loss value corresponding to the auxiliary encoder 2 is determined by calculating the third similarity distance and the fourth similarity distance corresponding to all negative samples in the negative sample queue 2, as well as other parameters in equation (1) above. The loss value corresponding to the auxiliary encoder 2 is characterized by the following equation (5):

[0084]

[0085] Where L2 is the loss value corresponding to co-encoder 2, p is the second image feature, ω2 is the first image feature output by co-encoder 2, and ω m Let sim(p,ω2) represent the image features corresponding to the m-th negative sample in negative sample queue 2, and let sim(p,ω2) represent the third similarity distance. m ) represents the fourth similarity distance corresponding to the m-th negative sample in negative sample queue 2, τ2 represents the temperature coefficient corresponding to the co-encoder 2, τ2 can be 0.9, and M is the number of negative samples in negative sample queue 2, M can be 65536.

[0086] After obtaining the loss values ​​corresponding to co-encoder 1 and co-encoder 2 respectively, the total loss value of co-encoder 1 and co-encoder 2 is determined based on the preset weights corresponding to co-encoder 1 and co-encoder 2 respectively, and the total loss value is represented by the following formula (6):

[0087] L=β1L1+β2L2(6)

[0088] Where L represents the total loss value, β1 is the weight corresponding to co-encoder 1, β2 is the weight corresponding to co-encoder 2, L1 is the loss value corresponding to co-encoder 1, and L2 is the loss value corresponding to co-encoder 2. β1 can be 0.55, and β2 can be 0.25.

[0089] Based on the obtained total loss value, the parameters of the master encoder are updated using the backpropagation algorithm.

[0090] Based on the updated parameters of the main encoder and equation (3) above, equation (7) is constructed to update the parameters of the auxiliary encoder 1 according to the updated parameters of the main encoder and using the momentum update algorithm of the corresponding auxiliary encoder 1:

[0091] θ1←α1θ1+(1-α1)θ p (7)

[0092] In equation (7) above, θ1 to the left of the arrow represents the updated parameters of the auxiliary encoder 1, and θ1 to the right of the arrow represents the original parameters of the auxiliary encoder 1. p The parameters are updated for the main encoder, and α1 is the momentum coefficient corresponding to the auxiliary encoder 1. α1 can be 0.999.

[0093] Based on the updated parameters of the main encoder and equation (3) above, equation (8) is constructed to update the parameters of the auxiliary encoder 2 according to the updated parameters of the main encoder and using the momentum update algorithm of the corresponding auxiliary encoder 2:

[0094] θ2←α2θ2+(1-α2)θ p (8)

[0095] In equation (8), θ2 to the left of the arrow represents the updated parameters of the co-encoder 2, and θ2 to the right of the arrow in equation (7) represents the original parameters of the co-encoder 2. p The parameters are the updated main encoder parameters, and α2 is the momentum coefficient corresponding to the auxiliary encoder 2. α2 can be 0.9.

[0096] pass Figure 2 The framework shown has two auxiliary encoders, which can improve the feature discrimination of positive and negative samples extracted by the main encoder without increasing the computational load. This makes it easier to improve the performance of the model corresponding to the downstream task when the pre-trained main encoder is applied to the downstream task. The model corresponding to the downstream task is obtained by fine-tuning the pre-trained main encoder.

[0097] The following explains the application of the pre-trained master encoder to downstream tasks. The trained master encoder includes a first network for extracting high-dimensional image features and a second network for dimensionality reduction of these features. After obtaining the trained master encoder, the method may further include: constructing an image processing model using a decoder and the trained master encoder; and training the decoder and second network of the image processing model with labeled data while keeping the first network fixed until a second convergence condition is met, thus obtaining the trained image processing model.

[0098] It is worth noting that the first and second networks here have the same function as the first and second networks in the previous embodiments. The difference is that the first and second networks in this embodiment are the networks included in the master encoder that have been pre-trained.

[0099] In this context, labeled data can be understood as image data with labels.

[0100] The settings and functions of the decoder are related to the type of downstream task. For example, when the downstream task is a classification task, the decoder acts as a classifier to classify the target image; or, when the downstream task is an object detection task, the decoder acts as both a classifier and a regressor to classify and locate different objects in the target image.

[0101] Training the decoder and the second network means updating the parameters of the decoder and the second network based on the differences between the output of the image processing model and the data carried in the labeled data.

[0102] The second convergence condition can be that the difference between the output of the image processing model and the labeled data is less than or equal to a third preset threshold, or the second convergence condition can be that the number of times the parameters of the decoder and the parameters of the second network are updated reaches a fourth preset threshold.

[0103] The labeled data includes multiple data sets. During the training of the decoder and the second network of the image processing model using the labeled data, resampling can be used in the labeled data to iteratively train the decoder and the second network of the image processing model until the second convergence condition is met, thus obtaining the trained image processing model.

[0104] By freezing the first network in the image processing model, i.e. fixing the first network, the network parameters in the first network are not adjusted during the training process of the image processing model. Instead, the parameters of the second network and the decoder are updated. This ensures that the high-dimensional features captured by the pre-trained main encoder do not change significantly during fine-tuning. This allows the pre-trained high-dimensional features to continue to be used effectively in new tasks without causing inappropriate impacts on them. In addition, the resampling method enables the model to have better robustness in tasks with long-tailed datasets.

[0105] Figure 3 This is a flowchart illustrating an image processing method according to an exemplary embodiment, with reference to... Figure 3 This includes the following steps:

[0106] Step 301: Obtain the target image;

[0107] Step 302: Process the target image according to the image processing model to obtain the image processing result. The image processing model is obtained by training the decoder and the main encoder.

[0108] It is worth noting that the master encoder here is the master encoder trained in the aforementioned embodiments.

[0109] The target image can be a medical image.

[0110] Specifically, when the image processing model is an image classification model, the image processing result corresponds to the category of the target image; when the image processing model is an object detection model, the image processing result corresponds to the category of different objects in the target image and the position of different objects in the target image.

[0111] The training process of the image processing model can be referred to the above-mentioned relevant embodiments, and will not be repeated here.

[0112] Figure 4 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment, with reference to... Figure 4 The device 400 includes the following modules:

[0113] The first acquisition module 401 is used to acquire the original image and the enhanced image obtained by performing image enhancement on the original image;

[0114] The output module 402 is used to input the enhanced image into at least two auxiliary encoders respectively to obtain the first image features output by each auxiliary encoder, and to input the original image into the main encoder to obtain the second image features output by the main encoder.

[0115] The learning module 403 is used to perform comparative learning on the main encoder based on the first image features, the second image features, and the acquired negative samples to obtain a trained main encoder. The trained main encoder is used to extract image features of the target image. During the comparative learning process, the backpropagation algorithm is used to update the parameters of the main encoder, and the momentum update algorithm is used to update the parameters of each of the auxiliary encoders. The momentum coefficients used in the momentum update algorithm corresponding to each auxiliary encoder are different.

[0116] In some embodiments, the learning module 403 includes:

[0117] The first determining submodule is used to determine the loss value corresponding to each of the auxiliary encoders based on the negative samples in the negative sample queue corresponding to the auxiliary encoder, the first image feature corresponding to the auxiliary encoder, and the second image feature.

[0118] The update submodule is used to update the parameters of the main encoder using the backpropagation algorithm based on all the loss values, and to update the parameters of each auxiliary encoder using the momentum update algorithm corresponding to each auxiliary encoder based on the updated parameters of the main encoder, until the first convergence condition is met, and the trained main encoder is obtained.

[0119] In some embodiments, the first determining submodule is specifically used to: for each of the auxiliary encoders, based on the negative samples in the negative sample queue corresponding to the auxiliary encoder, the first image feature corresponding to the auxiliary encoder, and the second image feature, determine the loss value corresponding to the auxiliary encoder using the following formula:

[0120]

[0121] Among them, L i Let p be the loss value corresponding to the i-th co-encoder, p be the second image feature, and ω be the loss value. i ω represents the first image feature output by the i-th co-encoder. n For the nth negative sample in the negative sample queue corresponding to the i-th co-encoder, τ i The preset temperature coefficient is the one corresponding to the i-th auxiliary encoder, N is the number of negative samples in the negative sample queue corresponding to the i-th auxiliary encoder, exp() is the exponential function with the natural constant e as the base, sim() is the similarity function, and log() is the logarithmic function.

[0122] In some embodiments, the update submodule is specifically used for:

[0123] All the aforementioned loss values ​​are weighted according to preset weights to obtain the total loss value;

[0124] Based on the total loss value, the parameters of the master encoder are updated using the backpropagation algorithm, and the parameters of each auxiliary encoder are updated using the momentum update algorithm corresponding to each auxiliary encoder based on the updated parameters of the master encoder, until the first convergence condition is met, and the trained master encoder is obtained.

[0125] In some embodiments, the trained master encoder includes a first network for extracting high-dimensional features of an image and a second network for performing dimensionality reduction processing on the high-dimensional features. The apparatus 400 further includes:

[0126] The building block is used to construct an image processing model using the decoder and the trained master encoder;

[0127] The training module is used to train the decoder of the image processing model and the second network with labeled data in a fixed manner, until the second convergence condition is met, so as to obtain the trained image processing model.

[0128] The implementation methods of each module in the above-mentioned device 400 can be referred to the above-mentioned related embodiments, and will not be repeated here.

[0129] Figure 5This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment, with reference to... Figure 5 The device 500 includes the following modules:

[0130] The second acquisition module 501 is used to acquire the target image;

[0131] The processing module 502 is used to process the target image according to the image processing model to obtain the image processing result, wherein the image processing model is obtained by training the decoder and the trained master encoder.

[0132] The implementation methods of each module in the above-mentioned device 500 can be referred to the above-mentioned related embodiments, and will not be repeated here.

[0133] This disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the above-described model training method or image processing method.

[0134] This disclosure also provides an electronic device, including:

[0135] A storage device on which computer programs are stored;

[0136] A processing device is used to execute the computer program in the storage device to implement the steps of the above-described model training method or image processing method.

[0137] The following is for reference. Figure 6 This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0138] like Figure 6As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0139] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0140] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0141] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0142] In some implementations, electronic devices can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0143] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0144] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire an original image and an enhanced image obtained by image enhancement of the original image; input the enhanced image into at least two auxiliary encoders respectively to obtain first image features output by each auxiliary encoder, and input the original image into a main encoder to obtain second image features output by the main encoder; perform comparative learning on the main encoder based on each of the first image features, the second image features, and the acquired negative samples to obtain a trained main encoder, which is used to extract image features of the target image. During the comparative learning process, the parameters of the main encoder are updated using a backpropagation algorithm, and the parameters of each auxiliary encoder are updated using a momentum update algorithm. The momentum coefficients used in the momentum update algorithms corresponding to each auxiliary encoder are different.

[0145] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a target image; process the target image according to an image processing model to obtain an image processing result, wherein the image processing model is obtained by training a decoder and a trained master encoder.

[0146] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0149] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0150] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (C...

[0151] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0152] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0153] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A model training method, characterized in that, include: Acquire the original image and the enhanced image obtained by performing image enhancement on the original image; The enhanced image is input into at least two auxiliary encoders to obtain the first image features output by each auxiliary encoder, and the original image is input into the main encoder to obtain the second image features output by the main encoder. The master encoder is subjected to comparative learning based on the first image features, the second image features, and the obtained negative samples to obtain the trained master encoder. The trained master encoder is used to extract image features of the target image. During the comparative learning process, the backpropagation algorithm is used to update the parameters of the master encoder, and the momentum update algorithm is used to update the parameters of each of the auxiliary encoders. The momentum coefficients used in the momentum update algorithm corresponding to each auxiliary encoder are different. The step of performing comparative learning on the master encoder based on the first image features, the second image features, and the acquired negative samples to obtain the trained master encoder includes: for each of the auxiliary encoders, determining the loss value corresponding to the auxiliary encoder based on the negative samples in the acquired negative sample queue corresponding to the auxiliary encoder, the first image features corresponding to the auxiliary encoder, and the second image features; updating the parameters of the master encoder using a backpropagation algorithm based on all the loss values, and updating the parameters of each auxiliary encoder using a momentum update algorithm corresponding to each of the auxiliary encoders based on the updated parameters of the master encoder, until the first convergence condition is met, thereby obtaining the trained master encoder.

2. The method according to claim 1, characterized in that, For each of the auxiliary encoders, the loss value corresponding to that auxiliary encoder is determined based on the negative samples in the negative sample queue corresponding to that auxiliary encoder, the first image feature corresponding to that auxiliary encoder, and the second image feature, including: For each of the aforementioned co-encoders, based on the negative samples in the negative sample queue corresponding to that co-encoder, the first image feature corresponding to that co-encoder, and the second image feature, the loss value corresponding to that co-encoder is determined using the following formula: ； in, For the loss value corresponding to the i-th co-encoder, The second image feature, The first image feature output by the i-th co-encoder. This is the nth negative sample in the negative sample queue corresponding to the i-th auxiliary encoder. The preset temperature coefficient is the one corresponding to the i-th auxiliary encoder, N is the number of negative samples in the negative sample queue corresponding to the i-th auxiliary encoder, exp() is the exponential function with the natural constant e as the base, sim() is the similarity function, and log() is the logarithmic function.

3. The method according to claim 1, characterized in that, The step of updating the parameters of the master encoder using a backpropagation algorithm based on all the loss values, and then updating the parameters of each of the auxiliary encoders using a momentum update algorithm corresponding to each of the auxiliary encoders based on the updated parameters of the master encoder, until a first convergence condition is met, to obtain the trained master encoder, includes: All the aforementioned loss values ​​are weighted according to preset weights to obtain the total loss value; Based on the total loss value, the parameters of the master encoder are updated using the backpropagation algorithm, and the parameters of each auxiliary encoder are updated using the momentum update algorithm corresponding to each auxiliary encoder based on the updated parameters of the master encoder, until the first convergence condition is met, and the trained master encoder is obtained.

4. The method according to any one of claims 1-3, characterized in that, The trained master encoder includes a first network for extracting high-dimensional features from the image and a second network for reducing the dimensionality of these high-dimensional features. The method further includes: An image processing model is constructed using the decoder and the trained master encoder; The image processing model's decoder and the second network are trained using labeled data while the first network is fixed, until the second convergence condition is met, thus obtaining the trained image processing model.

5. An image processing method, characterized in that, include: Acquire the target image; The target image is processed according to the image processing model to obtain the image processing result, wherein the image processing model is obtained by training the decoder and the master encoder trained according to any one of claims 1-4.

6. A model training device, characterized in that, include: The first acquisition module is used to acquire the original image and the enhanced image obtained by performing image enhancement on the original image; The output module is used to input the enhanced image into at least two auxiliary encoders respectively to obtain the first image features output by each auxiliary encoder, and to input the original image into the main encoder to obtain the second image features output by the main encoder. The learning module is used to perform comparative learning on the main encoder based on the first image features, the second image features, and the acquired negative samples to obtain the trained main encoder. The trained main encoder is used to extract image features of the target image. During the comparative learning process, the backpropagation algorithm is used to update the parameters of the main encoder, and the momentum update algorithm is used to update the parameters of each of the auxiliary encoders. The momentum coefficients used in the momentum update algorithm corresponding to each auxiliary encoder are different. The learning module includes: The first determining submodule is used to determine the loss value corresponding to each of the auxiliary encoders based on the negative samples in the negative sample queue corresponding to the auxiliary encoder, the first image feature corresponding to the auxiliary encoder, and the second image feature. The update submodule is used to update the parameters of the main encoder using the backpropagation algorithm based on all the loss values, and to update the parameters of each auxiliary encoder using the momentum update algorithm corresponding to each auxiliary encoder based on the updated parameters of the main encoder, until the first convergence condition is met, and the trained main encoder is obtained.

7. An image processing apparatus, characterized in that, include: The second acquisition module is used to acquire the target image; The processing module is used to process the target image according to the image processing model to obtain the image processing result, wherein the image processing model is obtained by training the decoder and the master encoder trained according to any one of claims 1-4.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processing device, the program implements the steps of the method according to any one of claims 1-4, or the steps of the method according to claim 5.

9. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing apparatus for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-4, or the steps of the method according to claim 5.

Images