Network training, image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a network training method and device, an image processing method and device, an electronic device and a storage medium. The network training method comprises: determining an image processing loss of an initial neural network based on support set images and query set images; determining self-supervised training data based on the support set images and / or the query set images; determining a self-supervised loss of the initial neural network based on the self-supervised training data; training the initial neural network based on the image processing loss and the self-supervised loss to obtain a target neural network; wherein the target neural network is used for image processing operation on a to-be-processed image to obtain a target image processing result. The embodiments of the present disclosure can comprehensively combine few-shot meta-learning and self-supervised learning to obtain a target neural network with high network performance.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and in particular, to a network training method and device, an image processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Few-shot learning is one of the important emerging research fields in computer vision. In recent years, due to the rapid development of convolutional neural networks (CNN), many image processing tasks based on CNN have achieved great success, such as image classification tasks, image detection tasks, and image segmentation tasks. However, the current CNN model is obviously dependent on a large amount of data, and if there is only a small amount of training data, the model will often appear overfitting. Few-shot learning aims to develop the ability of the CNN model in the few-shot scenario. SUMMARY

[0003] The present disclosure provides a network training method and device, an image processing method and device, an electronic device, and a storage medium.

[0004] According to an aspect of the present disclosure, a network training method is provided, including: determining an image processing loss of an initial neural network based on support set images and query set images; determining self-supervised training data based on the support set images and / or the query set images; determining a self-supervised loss of the initial neural network based on the self-supervised training data; training the initial neural network based on the image processing loss and the self-supervised loss to obtain a target neural network; wherein the target neural network is used to perform an image processing operation on a to-be-processed image to obtain a target image processing result.

[0005] In a possible implementation, the determining of the image processing loss of the initial neural network based on the support set images and the query set images includes: determining a predicted image processing result of the query set images based on the support set images and the initial neural network; and determining the image processing loss based on the predicted image processing result of the query set images and preset labels of the query set images.

[0006] In a possible implementation, the self-supervised loss includes a support set rotation angle prediction loss; the determining of the self-supervised training data based on the support set images includes: rotating the support set images, and determining the rotated support set images as the self-supervised training data; and the determining of the self-supervised loss of the initial neural network based on the self-supervised training data includes: determining a predicted rotation angle of the rotated support set images by using the initial neural network, and determining the support set rotation angle prediction loss based on the predicted rotation angle of the rotated support set images.

[0007] In a possible implementation, the self-supervised loss includes a query set rotation angle prediction loss; the determining of the self-supervised training data based on the query set image includes: rotating the query set image, and determining the rotated query set image as the self-supervised training data; and the determining of the self-supervised loss of the initial neural network based on the self-supervised training data includes: determining, by using the initial neural network, a predicted rotation angle of the rotated query set image, and determining the query set rotation angle prediction loss based on the predicted rotation angle of the rotated query set image.

[0008] In a possible implementation, the self-supervised loss includes a support set relative position prediction loss; the determining of the self-supervised training data based on the support set image includes: performing image block segmentation on the support set image, and determining a plurality of support set image blocks segmented from the support set image as the self-supervised training data; and the determining of the self-supervised loss of the initial neural network based on the self-supervised training data includes: determining, by using the initial neural network, a predicted relative position between any two support set image blocks, and determining the support set relative position prediction loss based on the predicted relative position between the any two support set image blocks.

[0009] In a possible implementation, the self-supervised loss includes a query set relative position prediction loss; the determining of the self-supervised training data based on the query set image includes: performing image block segmentation on the query set image, and determining a plurality of query set image blocks segmented from the query set image as the self-supervised training data; and the determining of the self-supervised loss of the initial neural network based on the self-supervised training data includes: determining, by using the initial neural network, a predicted relative position between any two query set image blocks, and determining the query set relative position prediction loss based on the predicted relative position between the any two query set image blocks.

[0010] In a possible implementation, the self-supervised loss includes a support set feature consistency loss; the determining of the self-supervised training data based on the support set image includes: performing two different image enhancement processing operations on the support set image, and determining two enhanced support set images obtained through the operations as the self-supervised training data; and the determining of the self-supervised loss of the initial neural network based on the self-supervised training data includes: respectively determining, by using the initial neural network, image features corresponding to each of the two enhanced support set images, and determining the support set feature consistency loss based on the image features corresponding to the two enhanced support set images.

[0011] In a possible implementation, the self-supervised loss comprises a query set feature consistency loss; the determining of the self-supervised training data based on the query set image comprises: performing two different image enhancement processing operations on the query set image, and determining the two enhanced query set images as the self-supervised training data; and the determining of the self-supervised loss of the initial neural network based on the self-supervised training data comprises: determining the image features corresponding to each enhanced query set image by using the initial neural network, and determining the query set feature consistency loss based on the image features corresponding to the two enhanced query set images.

[0012] In a possible implementation, the training of the initial neural network based on the image processing loss and the self-supervised loss to obtain a target neural network comprises: performing weighted summation on the image processing loss and the self-supervised loss to obtain a total loss; and adjusting network parameters of the initial neural network based on the total loss to obtain the target neural network.

[0013] According to an aspect of the present disclosure, an image processing method is provided, comprising: obtaining a to-be-processed image; and performing an image processing operation on the to-be-processed image by using a target neural network to obtain a target image processing result; wherein the target neural network is trained based on the above method.

[0014] According to an aspect of the present disclosure, a network training apparatus is provided, comprising: an image processing loss determination module configured to determine an image processing loss of an initial neural network based on a support set image and a query set image; a self-supervised training data determination module configured to determine self-supervised training data based on the support set image and / or the query set image; a self-supervised loss determination module configured to determine a self-supervised loss of the initial neural network based on the self-supervised training data; and a training module configured to train the initial neural network based on the image processing loss and the self-supervised loss to obtain a target neural network; wherein the target neural network is configured to perform an image processing operation on a to-be-processed image to obtain a target image processing result.

[0015] According to an aspect of the present disclosure, an image processing apparatus is provided, comprising: an obtaining module configured to obtain a to-be-processed image; and an image processing module configured to perform an image processing operation on the to-be-processed image by using a target neural network to obtain a target image processing result; wherein the target neural network is trained based on the above method.

[0016] According to an aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory configured to store processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.

[0017] According to an aspect of the present disclosure, a computer readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the above method.

[0018] In the embodiments of the present disclosure, in order to enable the initial neural network to have good performance after being trained on few samples, an image processing loss of the initial neural network is determined based on the corresponding support set images and query set images of few-shot meta-learning, in addition, in order to enable the initial neural network to have the performance of learning the hidden characteristics in the images after being trained, self-supervised training data is determined based on the support set images and / or the query set images, and a self-supervised loss of the initial neural network is determined based on the self-supervised training data, and then the initial neural network is trained by comprehensively considering the image processing loss and the self-supervised loss, so as to comprehensively consider few-shot meta-learning and self-supervised learning, and obtain a target neural network with high network performance, and the target neural network can be used for image processing operation on the to-be-processed images, and a target image processing result with high precision is obtained.

[0019] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the technical solutions of the present disclosure.

[0021] Figure 1 A flowchart of a network training method according to an embodiment of the present disclosure is shown;

[0022] Figure 2 A schematic diagram of an initial neural network according to an embodiment of the present disclosure is shown;

[0023] Figure 3 A flowchart of an image processing method according to an embodiment of the present disclosure is shown;

[0024] Figure 4 A block diagram of a network training device according to an embodiment of the present disclosure is shown;

[0025] Figure 5 A block diagram of an image processing device according to an embodiment of the present disclosure is shown;

[0026] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0027] Figure 7A block diagram illustrating another electronic device according to an embodiment of the disclosure is shown. DETAILED DESCRIPTION

[0028] Various exemplary embodiments, features and aspects of the present disclosure will be explained in detail below with reference to the accompanying drawings. The same reference numerals are used throughout the drawings and the same elements are represented by the same characters without redundant description. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0029] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0030] The term "and / or" used herein only means an association relationship of the associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0031] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some examples, methods, means, elements and circuits that are well known to those skilled in the art are not described in detail in order to highlight the main idea of the present disclosure.

[0032] Few-shot learning is one of the important emerging research fields in computer vision. In recent years, due to the rapid development of CNN, many CNN-based target image processing tasks have achieved great success, such as image classification tasks, image detection tasks, image segmentation tasks, etc. However, the current CNN model is obviously dependent on a large amount of data, and if there is only a small amount of training data, the model will often appear overfitting. Few-shot learning aims to develop the ability of CNN model in the scene of a small amount of samples.

[0033] And self-supervised learning is a model training method that does not rely on data labels, which can learn the hidden characteristics of images. This kind of method has been widely used in image classification, target detection and other image processing fields, and its purpose is mainly to provide an initialization model with good network performance based on self-supervised learning.

[0034] In the few-shot learning scenario in the image classification field, there are some application cases of self-supervised learning. For example, the rotation angle prediction, the contrast learning, and other self-supervised learning methods have been applied to the few-shot image classification model, and combined with the unlabeled image data to help the training of the few-shot image classification model. However, these model training methods based on self-supervised learning are basically applied to the technology route based on pre-training and fine-tuning in a specific image processing scenario (for example, the image classification scenario), and the model needs to be retrained when other image processing scenarios are transformed. For the common meta-learning method in few-shot learning, there is currently no model training method combined with self-supervised learning.

[0035] The network training method of the embodiments of the present disclosure can be applied to the meta-learning scenario common in few-shot learning, and can be used in the meta-learning scenario of few-shot image classification, few-shot image detection, few-shot image segmentation, and other few-shot image processing tasks.

[0036] In order to enable the initial neural network to have good performance after training on few-shot, based on the support set image and the query set image corresponding to the few-shot meta-learning, the image processing loss of the initial neural network is determined, in addition, in order to enable the initial neural network to have the performance of learning the hidden characteristics of the image after training, based on the support set image and / or the query set image, the self-supervised training data is determined, and based on the self-supervised training data, the self-supervised loss of the initial neural network is determined, and then the initial neural network is trained by integrating the image processing loss and the self-supervised loss, so as to integrate the few-shot meta-learning and the self-supervised learning, and obtain the target neural network with high network performance. Subsequently, the target neural network can be used to perform image processing operation on the image to be processed, and obtain the target image processing result with high precision.

[0037] The network training method provided by the embodiments of the present disclosure will be described in detail below.

[0038] Figure 1 A flowchart of a network training method according to an embodiment of the present disclosure is shown. The network training method can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The network training method can be realized by a processor calling computer readable instructions stored in a memory. Alternatively, the network training method can be executed by a server. As shown in the figure, the network training method can include: Figure 1

[0039] ​In step S11, based on the support set image and the query set image, an image processing loss in the initial neural network is determined.

[0040] The training process of the meta-learning needs two types of data sets: a support set and a query set, wherein the support set includes support set images, and the query set includes query set images.

[0041] The initial neural network can include an image processing branch corresponding to an image processing scene to which the initial neural network is applied.

[0042] In the case where the network training method is applied to a few-shot image classification scene, the image processing branch in the initial neural network is an image classification branch. At this time, the image processing loss is a classification loss.

[0043] In the case where the network training method is applied to a few-shot image detection scene, the image processing branch in the initial neural network is an image detection branch. At this time, the image processing loss is a classification loss and a regression loss.

[0044] In the case where the network training method is applied to a few-shot image segmentation scene, the image processing branch in the initial neural network is an image segmentation branch. At this time, the image processing loss is a classification loss and a regression loss.

[0045] The image processing branch in the initial neural network can also be set to a corresponding image processing branch according to an actual application image processing scene, which is not limited in the present disclosure.

[0046] In step S12, based on the support set image and / or the query set image, self-supervised training data is determined.

[0047] In order to enable the initial neural network to have the performance of learning the hidden features in the image after being trained, the support set image and / or the query set image is preprocessed to determine self-supervised training data for self-supervised training. The manner of determining the self-supervised training data will be described in detail in combination with possible implementation manners of the present disclosure, which will not be repeated here.

[0048] In step S13, based on the self-supervised training data, a self-supervised loss of the initial neural network is determined.

[0049] Compared with the meta-learning in the related art, the embodiment of the present disclosure adds a self-supervised branch in the initial neural network, so that on the basis of few-shot meta-learning, self-supervised learning is added, so that the initial neural network can have the performance of learning the hidden features in the image after being trained.

[0050] The self-supervised branch can include at least one of a rotation angle prediction branch, a relative position prediction branch, and a feature consistency branch, and other self-supervised branches capable of self-supervised learning can also be set according to actual conditions, and the present disclosure does not make specific limitations in this regard.

[0051] The process of determining the self-supervised loss of the initial neural network based on the self-supervised training data will be described in detail in the following in combination with possible implementation manners of the present disclosure, and will not be repeated here.

[0052] In step S14, the initial neural network is trained based on the image processing loss and the self-supervised loss to obtain a target neural network; wherein the target neural network is used to perform an image processing operation on the to-be-processed image to obtain a target image processing result.

[0053] After determining the image processing loss and the self-supervised loss, the initial neural network is trained by comprehensively combining the image processing loss and the self-supervised loss, so as to comprehensively combine the few-shot meta-learning and the self-supervised learning to obtain the target neural network with higher network performance.

[0054] The target neural network only includes the trained image processing branch, but does not include the trained self-supervised branch. The target image processing result with high accuracy can be obtained by using the target neural network to perform an image processing operation on the to-be-processed image.

[0055] In the embodiment of the present disclosure, in order to enable the initial neural network to have good performance after being trained on few samples, the image processing loss of the initial neural network is determined based on the support set image and the query set image corresponding to the few-shot meta-learning, in addition, in order to enable the initial neural network to have the performance of learning the hidden characteristics of the image after being trained, the self-supervised training data is determined based on the support set image and / or the query set image, and the self-supervised loss of the initial neural network is determined based on the self-supervised training data, and then the initial neural network is trained by comprehensively combining the image processing loss and the self-supervised loss, so as to comprehensively combine the few-shot meta-learning and the self-supervised learning to obtain the target neural network with higher network performance, and subsequently the target image processing result with high accuracy can be obtained by using the target neural network to perform an image processing operation on the to-be-processed image.

[0056] In a possible implementation manner, the image processing operation includes one of the following: an image classification operation, an image detection operation, and an image segmentation operation.

[0057] In the case where the network training method is applied to a few-shot image classification scenario, the target neural network can be used to perform an image classification operation on the to-be-processed image.

[0058] In the case where the network training method is applied to a few-shot image detection scenario, the target neural network can be used to perform an image detection operation on the to-be-processed image.

[0059] In the case where the network training method is applied to a few-shot image segmentation scenario, the target neural network can be used to perform an image segmentation operation on a to-be-processed image.

[0060] The process of network training on the initial neural network by integrating few-shot meta-learning and self-supervised learning will be described in detail below.

[0061] Figure 2 A schematic diagram of an initial neural network according to an embodiment of the present disclosure is shown. As shown in the figure, the initial neural network includes a backbone, image processing modules, and multiple self-supervised modules. Among them, as shown in the figure, Figure 2 The backbone is used for image feature extraction, the image processing modules are used for outputting image processing results, and the self-supervised modules are used for self-supervised training. Figure 2

[0062] The backbone is used for image feature extraction, the image processing modules are used for outputting image processing results, and the self-supervised modules are used for self-supervised training.

[0063] As shown in the figure, the backbone and the image processing modules jointly constitute an image processing branch, the backbone and the support set self-supervised module jointly constitute a support set self-supervised branch, and the backbone and the query set self-supervised module jointly constitute a query set self-supervised branch. Figure 2

[0064] Based on the image processing branch, the support set self-supervised branch, and the query set self-supervised branch in the initial neural network, the entire network training phase can be divided into multiple training tasks. For example, an image processing training task based on the image processing branch, a self-supervised training task based on the support set self-supervised branch, and a self-supervised training task based on the query set branch. Each training task can be executed simultaneously or sequentially, and the present disclosure does not make specific limitations on this.

[0065] The support set image and the query set image are input into Figure 2 The initial neural network shown in the figure, and based on the support set image and the query set image, an image processing training task can be performed to determine the image processing loss of the image processing branch in the initial neural network. ​​

[0066] In a possible implementation, the image processing loss of the initial neural network is determined based on the support set images and the query set images, including: determining a predicted image processing result of the query set images based on the support set images and the initial neural network; and determining the image processing loss based on the predicted image processing result of the query set images and preset labels of the query set images.

[0067] In the network training process, the predicted image processing result of the query set images is determined based on the support set images, and then the image processing loss of the image processing branch is determined based on the predicted image processing result of the query set images and the preset labels of the query set images.

[0068] In the case where the network training method is applied to a few-shot image classification scenario, the predicted image processing result of the query set images is a classification result of the query set images, and the image processing loss of the image processing branch is a classification loss.

[0069] In the case where the network training method is applied to a few-shot image detection scenario, the predicted image processing result of the query set images is a detection result of the query set images, and the image processing loss of the image processing branch is a classification loss and a regression loss.

[0070] In the case where the network training method is applied to a few-shot image segmentation scenario, the predicted image processing result of the query set images is a segmentation result of the query set images, and the image processing loss of the image processing branch is a classification loss and a regression loss.

[0071] The specific process of determining the image processing loss of the image processing branch in the initial neural network based on the support set images and the query set images can also refer to the specific training process of few-shot meta-learning in related technologies, and the present disclosure does not make specific limitations thereto.

[0072] The support set images are input into the initial neural network as shown, and the support set self-supervised branch can be trained based on the support set images. Figure 2 The support set self-supervised loss of the support set self-supervised branch can be determined based on the support set images.

[0073] In a possible implementation, the self-supervised loss includes a support set rotation angle prediction loss; the self-supervised training data is determined based on the support set images, including: rotating the support set images, and determining the rotated support set images as the self-supervised training data; and the self-supervised loss of the initial neural network is determined based on the self-supervised training data, including: determining a predicted rotation angle of the rotated support set images by using the initial neural network, and determining the support set rotation angle prediction loss based on the predicted rotation angle of the rotated support set images.

[0074] The support set self-supervised branch can include a support set rotation angle prediction branch to perform a self-supervised training task of support set rotation angle prediction.

[0075] For example, as shown in the above Figure 2 For example, as shown in the above Figure 2 The support set self-supervised module includes a support set rotation angle prediction module (Rot Head), and the backbone network and the support set rotation angle prediction module (Rot Head) together constitute the support set rotation angle prediction branch.

[0076] The support set image is rotated by a first preset angle to obtain a rotated support set image. The first preset angle can be determined according to actual conditions, for example, the first preset angle can be 0 degrees, 90 degrees, 180 degrees, 270 degrees, etc., and the present disclosure does not make specific limitations on this.

[0077] The rotated support set image is input into the support set rotation angle prediction branch as self-supervised training data. The backbone network extracts features from the rotated support set image, and inputs the extracted image features into the support set rotation angle prediction module (Rot Head) to output a predicted rotation angle of the rotated support set image.

[0078] The support set rotation angle prediction module (Rot Head) can be a linear classifier, and its output can be a 1-dimensional 4-bit vector indicating the predicted rotation angle of the rotated support set image.

[0079] For example, the output of the linear classifier is 1000, indicating that the predicted rotation angle of the support set image is 0 degrees; the output of the linear classifier is 0100, indicating that the predicted rotation angle of the support set image is 90 degrees; the output of the linear classifier is 0010, indicating that the predicted rotation angle of the support set image is 180 degrees; and the output of the linear classifier is 0001, indicating that the predicted rotation angle of the support set image is 360 degrees.

[0080] In addition to being a linear classifier, the support set rotation angle prediction module (Rot Head) can also be set to other specific structures according to actual conditions, and the present disclosure does not make specific limitations on this.

[0081] After determining the predicted rotation angle of the rotated support set image, the support set rotation angle prediction loss (Rot loss) can be determined by determining the difference between the predicted rotation angle of the rotated support set image and the first preset angle of the support set image rotation.

[0082] The difference between the predicted rotation angle of the support set image after rotation and the first preset angle can be determined by using a cross-entropy loss function to determine the support set rotation angle prediction loss. Other loss functions can also be used, which are not limited in the present disclosure.

[0083] In a possible implementation, the self-supervised loss includes a support set relative position prediction loss; the self-supervised training data is determined based on the support set image, including: performing image block segmentation on the support set image, and determining the plurality of support set image blocks segmented as the self-supervised training data; the self-supervised loss of the initial neural network is determined based on the self-supervised training data, including: determining the predicted relative position between any two support set image blocks by using the initial neural network, and determining the support set relative position prediction loss based on the predicted relative position between any two support set image blocks.

[0084] The support set self-supervised branch can include a support set relative position prediction branch to perform the self-supervised training task of support set relative position prediction.

[0085] The above Figure 2 For example, as Figure 2 shown, the support set self-supervised module includes a support set relative position prediction module (Rel Head), and the backbone network (Backbone) and the support set relative position prediction module (Rel Head) together constitute a support set relative position prediction branch.

[0086] The support set image is segmented into image blocks to obtain a plurality of support set image blocks. For example, the support set image blocks are segmented into 3*3 support set image blocks, and each support set image is labeled with 0-8. The specific number of support set image blocks can be determined according to actual conditions, which are not limited in the present disclosure.

[0087] Any two support set image blocks are selected as self-supervised training data and input into the support set relative position prediction branch. The backbone network (Backbone) extracts features from the selected any two support set image blocks, and inputs the extracted image features into the support set relative position prediction module (Rel Head) to output the predicted relative position between the selected any two support set image blocks.

[0088] The support set relative position prediction module (Rel Head) can be a linear classifier, and its output can be a 1-dimensional 9-bit vector, which is used to indicate the predicted relative position between the selected any two support set image blocks.

[0089] For example, the output of the linear classifier is 010000000, indicating that the difference between the labels of any two selected support set image blocks is 1, i.e., the selected any two support set image blocks are two support set image blocks labeled as 0 and 1, or two support set image blocks labeled as 3 or 4, or other two support set image blocks with a label difference of 1; the output of the linear classifier is 000100000, indicating that the difference between the labels of any two selected support set image blocks is 3, i.e., the selected any two support set image blocks are two support set image blocks labeled as 0 and 3, or two support set image blocks labeled as 2 or 5, or other two support set image blocks with a label difference of 3. By analogy, no further description is given.

[0090] In addition to being a linear classifier, the support set relative position prediction module (Rel Head) can also be set to other specific structures according to actual conditions, which are not specifically limited by the present disclosure.

[0091] After determining the predicted relative position between any two support set image blocks, the support set relative position prediction loss (Rel loss) can be determined by determining the difference between the predicted relative position between the any two support set image blocks and the actual relative position between the any two support set image blocks.

[0092] The difference between the predicted relative position between the any two support set image blocks and the actual relative position between the any two support set image blocks can be determined by using a cross-entropy loss function to determine the support set relative position prediction loss, and other loss functions can also be used, which are not specifically limited by the present disclosure.

[0093] In one possible implementation, the self-supervised loss includes a support set feature consistency loss; the self-supervised training data is determined based on the support set image, including: performing two different image enhancement processing operations on the support set image, and determining the two enhanced support set images obtained as the self-supervised training data; the self-supervised loss of the initial neural network is determined based on the self-supervised training data, including: determining the image features corresponding to each enhanced support set image by using the initial neural network, respectively, and determining the support set feature consistency loss based on the image features corresponding to the two enhanced support set images.

[0094] The support set self-supervised branch can include a support set feature consistency branch to perform a self-supervised training task of support set feature consistency.

[0095] The above Figure 2 For example, as Figure 2As shown, the support set self-supervised module includes the support set feature consistency module (InvHead), the backbone network (Backbone), and the support set feature consistency module (InvHead), which together constitute the support set feature consistency branch.

[0096] Two different image enhancement operations are performed on the support set image to obtain two enhanced support set images. The image enhancement operations may include random cropping (e.g., randomly cropping into an 84*84 region), color perturbation, horizontal inversion, image matting, etc., which are not specifically limited in this disclosure.

[0097] Two augmented support set images are used as self-supervised training data and input into the support feature consistency branch. The backbone network extracts features from the rotated support set images and inputs the extracted image features into the support set feature consistency module (Inv Head) to output the final image features for each augmented support set image.

[0098] The support set feature consistency module (Inv Head) can be a linear layer, or it can be set to other specific structures according to the actual situation. This disclosure does not make any specific limitations on this.

[0099] After determining the final image features of each augmented support set image, the support set feature consistency loss (Inv loss) can be determined by assessing the feature consistency between two final image features.

[0100] In this process, a feature-invariant loss function can be used to determine the feature consistency between two final image features, thereby determining the support set feature consistency loss. The feature-invariant loss function can include distance metrics based on the L2 norm, distance metrics based on cosine similarity, contrastive learning functions such as BYOL or SimCLR, and other feature-invariant loss functions can also be used; this disclosure does not specifically limit the specific use of these functions.

[0101] The self-supervised branches of the support set may include one or more of the following: support set rotation angle prediction branch, support set relative position prediction branch, and support set feature consistency branch. The self-supervised training of each branch can be carried out simultaneously or sequentially. This disclosure does not make any specific limitations on this.

[0102] Input the query set image Figure 2 After the initial neural network shown, a self-supervised training task for the query set self-supervised branch can be performed based on the query set image to determine the query set self-supervised loss of the query set self-supervised branch.

[0103] In a possible implementation, the self-supervised loss includes a query set rotation angle prediction loss; the self-supervised training data is determined based on the query set image, including: rotating the query set image, and determining the rotated query set image as the self-supervised training data; and the self-supervised loss of the initial neural network is determined based on the self-supervised training data, including: determining, by using the initial neural network, a predicted rotation angle of the rotated query set image, and determining the query set rotation angle prediction loss based on the predicted rotation angle of the rotated query set image.

[0104] The query set self-supervised branch can include a query set rotation angle prediction branch to perform the self-supervised training task of query set rotation angle prediction.

[0105] For example, as shown in the above Figure 2 , the query set self-supervised module includes a query set rotation angle prediction module (Rot Head), and the backbone network and the query set rotation angle prediction module (Rot Head) together constitute the query set rotation angle prediction branch. Figure 2 The self-supervised training process of the query set rotation angle prediction branch is similar to the self-supervised training process of the support set rotation angle prediction branch, which is not repeated here.

[0106] In a possible implementation, the self-supervised loss includes a query set relative position prediction loss; the self-supervised training data is determined based on the query set image, including: performing image block segmentation on the query set image, and determining a plurality of query set image blocks segmented from the query set image as the self-supervised training data; and the self-supervised loss of the initial neural network is determined based on the self-supervised training data, including: determining, by using the initial neural network, a predicted relative position between any two query set image blocks, and determining the query set relative position prediction loss based on the predicted relative position between any two query set image blocks.

[0107] The query set self-supervised branch can include a query set relative position prediction branch to perform the self-supervised training task of query set relative position prediction.

[0108] For example, as shown in the above

[0109] , the query set self-supervised module includes a query set relative position prediction module (Rel Head), and the backbone network and the query set relative position prediction module (Rel Head) together constitute the query set relative position prediction branch. Figure 2 Figure 2 The self-supervised training process of the query set relative position prediction branch is similar to the self-supervised training process of the support set relative position prediction branch, which is not repeated here.

[0110] The self-supervised training process of the query set relative position prediction branch is similar to the self-supervised training process of the support set relative position prediction branch, which is not repeated here. ​

[0111] In a possible implementation, the self-supervised loss includes a query set feature consistency loss; the self-supervised training data is determined based on the query set image, including: performing two different image enhancement processing operations on the query set image, and determining the two enhanced query set images as the self-supervised training data; and the self-supervised loss of the initial neural network is determined based on the self-supervised training data, including: determining the image features corresponding to each enhanced query set image by using the initial neural network, and determining the query set feature consistency loss based on the image features corresponding to the two enhanced query set images.

[0112] The query set self-supervised branch can include a query set feature consistency branch to perform a self-supervised training task of query set feature consistency.

[0113] Taking the above Figure 2 For example, as shown in Figure 3 The query set self-supervised module includes a query set feature consistency module (InvHead), and the backbone network (Backbone) and the query set feature consistency module (InvHead) together constitute the query set feature consistency branch.

[0114] The self-supervised training process of the query set feature consistency branch is similar to the self-supervised training process of the support set feature consistency branch described above, and is not repeated here.

[0115] The query set self-supervised branch can include one or more of a query set rotation angle prediction branch, a query set relative position prediction branch, and a query set feature consistency branch, and the self-supervised training of each branch can be performed simultaneously or sequentially, and the present disclosure does not make specific limitations thereon.

[0116] Compared with the self-supervised training of the support set image or the query set image in the related art, the embodiments of the present disclosure perform self-supervised training on both the support set image and the query set image, so that the sample amount of self-supervised training can be expanded, the network training effect can be improved, and the initial neural network can have stronger performance of learning the hidden characteristics of the image after training.

[0117] In a possible implementation, the initial neural network is trained based on the image processing loss and the self-supervised loss to obtain a target neural network, including: performing weighted summation on the image processing loss and the self-supervised loss to obtain a total loss; and adjusting the network parameters of the initial neural network based on the total loss to obtain the target neural network.

[0118] After the image processing loss and the self-supervised loss are determined respectively, the image processing loss and the self-supervised loss can be weighted and summed to obtain a total loss, and then based on the total loss, the network parameters of the initial neural network are adjusted to complete one network training. The above network training process is repeatedly iterated until a preset iteration condition is met, the training is ended, and the trained initial neural network is obtained.

[0119] The specific values of the weights of the image processing loss and the self-supervised loss can be set according to actual conditions, and the present disclosure does not make specific limitations thereto.

[0120] The preset iteration condition can be set according to actual conditions, for example, a preset number of iterations, a total loss reaching convergence, etc., and the present disclosure does not make specific limitations thereto.

[0121] Based on the trained image processing branch in the trained initial neural network, a target neural network is obtained, that is, the target neural network does not include the trained self-supervised branch.

[0122] Using the target neural network, an image processing operation can be performed on the to-be-processed image to obtain a target image processing result with high precision.

[0123] In the case where the network training method is applied to a few-shot image classification scenario, the target neural network can be used for image classification operation on the to-be-processed image to obtain a classification result with high precision.

[0124] In the case where the network training method is applied to a few-shot image detection scenario, the target neural network can be used for image detection operation on the to-be-processed image to obtain a detection result with high precision.

[0125] In the case where the network training method is applied to a few-shot image segmentation scenario, the target neural network can be used for image segmentation operation on the to-be-processed image to obtain a segmentation result with high precision.

[0126] The network training method of the embodiments of the present disclosure can be applied to other image processing scenarios in addition to the few-shot image classification scenario, the few-shot image detection scenario, and the few-shot image segmentation scenario, and the present disclosure does not make specific limitations thereto.

[0127] Figure 3A flowchart of an image processing method according to an embodiment of the present disclosure is shown. The image processing method can be performed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The image processing method can be implemented by a processor invoking computer-readable instructions stored in a memory. Alternatively, the image processing method can be performed by a server. As shown in Figure 4 The image processing method can include the following steps.

[0128] In step S31, a to-be-processed image is obtained.

[0129] In step S32, a target neural network is used to perform an image processing operation on the to-be-processed image to obtain a target image processing result.

[0130] The target neural network is trained based on the network training method described in the above embodiments.

[0131] In the case where the target neural network is trained based on the network training method described in the above embodiments in a few-shot image classification scenario, the target neural network can be used to perform an image classification operation on the to-be-processed image to obtain a classification result with high precision.

[0132] In the case where the target neural network is trained based on the network training method described in the above embodiments in a few-shot image detection scenario, the target neural network can be used to perform an image detection operation on the to-be-processed image to obtain a detection result with high precision.

[0133] In the case where the target neural network is trained based on the network training method described in the above embodiments in a few-shot image segmentation scenario, the target neural network can be used to perform an image segmentation operation on the to-be-processed image to obtain a segmentation result with high precision.

[0134] It can be understood that the above various method embodiments mentioned in the present disclosure can be combined with each other to form combined embodiments without violating the principle logic. Limited by the length, the present disclosure will not be repeated. It can be understood by those skilled in the art that the specific execution order of each step in the above method should be determined according to its function and possible internal logic.

[0135] Further, the present disclosure also provides a network training / image processing apparatus, an electronic device, a computer readable storage medium, and a program, which can be used to implement any of the network training / image processing methods provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding descriptions in the method part, and will not be repeated here.

[0136] Figure 4 A block diagram of a network training apparatus according to an embodiment of the present disclosure is shown. As shown, the apparatus 40 comprises: Figure 5

[0137] An image processing loss determination module 41 is configured to determine an image processing loss of the initial neural network based on the support set images and the query set images.

[0138] A self-supervised training data determination module 42 is configured to determine self-supervised training data based on the support set images and / or the query set images.

[0139] A self-supervised loss determination module 43 is configured to determine a self-supervised loss of the initial neural network based on the self-supervised training data.

[0140] A training module 44 is configured to train the initial neural network based on the image processing loss and the self-supervised loss to obtain a target neural network.

[0141] The target neural network is configured to perform an image processing operation on a to-be-processed image to obtain a target image processing result.

[0142] In a possible implementation, the image processing loss determination module 41 is specifically configured to:

[0143] determine a predicted image processing result of the query set images based on the support set images and the initial neural network;

[0144] determine the image processing loss based on the predicted image processing result of the query set images and preset labels of the query set images.

[0145] In a possible implementation, the self-supervised loss comprises a support set rotation angle prediction loss.

[0146] The self-supervised training data determination module 42 is specifically configured to:

[0147] rotate the support set images, and determine the rotated support set images as the self-supervised training data.

[0148] The self-supervised loss determination module 43 is specifically configured to:

[0149] determine a predicted rotation angle of the rotated support set images by using the initial neural network, and determine the support set rotation angle prediction loss based on the predicted rotation angle of the rotated support set images. ​

[0150] In a possible implementation, the self-supervised loss includes a query set rotation angle prediction loss;

[0151] The self-supervised training data determination module 42 is specifically configured to:

[0152] perform rotation on the query set image, and determine the rotated query set image as the self-supervised training data;

[0153] The self-supervised loss determination module 43 is specifically configured to:

[0154] determine, by using the initial neural network, a predicted rotation angle of the rotated query set image, and determine the query set rotation angle prediction loss based on the predicted rotation angle of the rotated query set image.

[0155] In a possible implementation, the self-supervised loss includes a support set relative position prediction loss;

[0156] The self-supervised training data determination module 42 is specifically configured to:

[0157] perform image block segmentation on the support set image, and determine a plurality of support set image blocks segmented from the support set image as the self-supervised training data;

[0158] The self-supervised loss determination module 43 is specifically configured to:

[0159] determine, by using the initial neural network, a predicted relative position between any two support set image blocks, and determine the support set relative position prediction loss based on the predicted relative position between any two support set image blocks.

[0160] In a possible implementation, the self-supervised loss includes a query set relative position prediction loss;

[0161] The self-supervised training data determination module 42 is specifically configured to:

[0162] perform image block segmentation on the query set image, and determine a plurality of query set image blocks segmented from the query set image as the self-supervised training data;

[0163] The self-supervised loss determination module 43 is specifically configured to:

[0164] determine, by using the initial neural network, a predicted relative position between any two query set image blocks, and determine the query set relative position prediction loss based on the predicted relative position between any two query set image blocks.

[0165] In a possible implementation, the self-supervised loss includes a support set feature consistency loss;

[0166] The self-supervised training data determination module 42 is specifically configured to:

[0167] two different image enhancement processing operations are performed on the support set image, and the two enhanced support set images obtained are determined as the self-supervised training data;

[0168] The self-supervised loss determination module 43 is specifically configured to:

[0169] The initial neural network is used to determine the image features corresponding to each of the two enhanced support set images, respectively, and the support set feature consistency loss is determined based on the image features corresponding to the two enhanced support set images.

[0170] In a possible implementation, the self-supervised loss includes a query set feature consistency loss.

[0171] The self-supervised training data determination module 42 is specifically configured to:

[0172] Two different image enhancement processing operations are performed on the query set image, and the two enhanced query set images obtained are determined as the self-supervised training data.

[0173] The self-supervised loss determination module 43 is specifically configured to:

[0174] The initial neural network is used to determine the image features corresponding to each of the two enhanced query set images, respectively, and the query set feature consistency loss is determined based on the image features corresponding to the two enhanced query set images.

[0175] In a possible implementation, the training module 44 is specifically configured to:

[0176] The image processing loss and the self-supervised loss are weighted and summed to obtain a total loss.

[0177] Based on the total loss, the network parameters of the initial neural network are adjusted to obtain a target neural network.

[0178] Figure 5 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown. As shown in the figure, the apparatus 50 comprises: Figure 6

[0179] The acquisition module 51 is configured to acquire a to-be-processed image.

[0180] The image processing module 52 is configured to perform an image processing operation on the to-be-processed image by using a target neural network to obtain a target image processing result.

[0181] The target neural network is trained by the network training method described in the above embodiments.

[0182] ​The method has specific technical correlation with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware operation efficiency or execution effect (including reducing data storage, reducing data transmission, improving hardware processing speed, etc.), so as to obtain the technical effect of improving the internal performance of the computer system in line with the natural law.

[0183] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to execute the methods described in the above method embodiments, and specific implementations can refer to the descriptions of the above method embodiments. For brevity, they will not be repeated here.

[0184] The embodiments of the present disclosure also provide a computer-readable storage medium having computer program instructions stored therein, and the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0185] The embodiments of the present disclosure also provide an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

[0186] The embodiments of the present disclosure also provide a computer program product, including computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device executes the above method.

[0187] The electronic device can be provided as a terminal, a server or other forms of devices.

[0188] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Referring to Figure 6 , the electronic device 800 can be a terminal device such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.

[0189] Referring to Figure 7 , the electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0190] The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete the steps of the methods described above, in whole or in part. Moreover, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0191] The memory 804 is configured to store various types of data to support the operations of the electronic device 800. Examples of these data include instructions to operate any applications or methods on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be realized by any type of volatile or non-volatile memory devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disc, or optical disc.

[0192] The power component 806 provides power to the various components of the electronic device 800. The power component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0193] The multimedia component 808 includes a screen to provide an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a back camera. The front camera and / or the back camera can receive external multimedia data when the electronic device 800 is in an operating mode, such as a shooting mode or a video mode. Each of the front camera and the back camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0194] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive an external audio signal when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0195] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can include a keypad, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0196] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the electronic device 800. For example, the sensor component 814 can detect an open / closed position of the electronic device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change of location of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and a temperature change of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 814 can also include a light sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, useful for image- and / or light- capturing applications. In some embodiments, the sensor component 814 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0197] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a corresponding communication standard, such as a wireless fidelity (Wi-Fi), a second generation (2G) mobile communication technology, a third generation (3G) mobile communication technology, a fourth generation (4G) mobile communication technology, a long term evolution (LTE) of a universal mobile telecommunications technology, a fifth generation (5G) mobile communication technology, or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcasting management system via a broadcasting channel. In an example embodiment, the communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technology.

[0198] In an exemplary embodiment, the electronic device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements for performing the above-described methods.

[0199] In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804 including computer program instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to complete the above-described methods.

[0200] The present disclosure relates to the field of augmented reality, by acquiring image information of a target object in a real environment, and then detecting or recognizing the relevant features, states and attributes of the target object by means of various visual-related algorithms, so as to obtain an AR effect combining virtual and real that matches specific applications. Exemplarily, the target object can involve a face, a limb, a gesture, an action, etc. related to a human body, or a marker, a sign, etc. related to an object, or a sand table, a display area or a display item, etc. related to a venue or a place. The visual-related algorithms can involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of an object, pose or depth detection of an object, etc. The specific applications can not only involve interactive scenarios such as touring, navigation, explanation, reconstruction, virtual effect superimposed display, etc. related to a real scene or an object, but also interactive scenarios such as special effect processing related to a person, such as makeup beautification, limb beautification, special effect display, virtual model display, etc. The detection or recognition of the relevant features, states and attributes of the target object can be achieved by a convolutional neural network. The convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0201] Figure 7 A block diagram of another electronic device according to embodiments of the present disclosure is shown. Referring to Figure 7 , the electronic device 1900 can be provided as a server or a terminal device. Referring to ​ , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described methods.

[0202] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Microsoft Windows Server TM , Apple's graphical user interface-based operating system (Mac OSX TM ), a multi-user multi-processing computer operating system (Unix TM ), a free and open-source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ), or the like.

[0203] In an exemplary embodiment, there is also provided a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above-described method.

[0204] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0205] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0206] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0207] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0208] The computer readable program instructions can also be loaded onto a computing / processing device, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computing / processing device, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computing / processing device, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0209] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer readable storage medium having no data signals on it. The instructions can be executed by one or more processors of a computer or other programmable data processing apparatus to produce a computer implemented process such that the instructions, which execute via the one or more processors of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0210] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer readable storage medium having no data signals on it. The instructions can be executed by one or more processors of a computer or other programmable data processing apparatus to produce a computer implemented process such that the instructions, which execute via the one or more processors of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0211] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0212] The computer program product can be embodied by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) or the like.

[0213] The above description of the various embodiments is intended to be illustrative and not restrictive. Many other embodiments will be obvious to those of skill in the art upon reviewing the above description, and the general principles described herein can be applied to other embodiments. Thus, the description is to be regarded as illustrating the principles described herein and not as limiting the scope of the inventive embodiments. The illustrations presented herein are not meant to be actual views of any particular apparatus but are merely ideal representations which emphasize related functions over other features.

[0214] Those skilled in the art can understand that, in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined according to its function and possible internal logic.

[0215] If the technical solution of the present application involves personal information, the product applying the technical solution of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solution of the present application involves sensitive personal information, the product applying the technical solution of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that the personal information collection range has been entered and the personal information will be collected. If the person voluntarily enters the collection range, it is regarded as agreeing to collect the personal information. Or, on the device for processing personal information, the personal information processing rules are informed by using obvious marks / information, and the personal authorization is obtained by means of pop-up information or asking the person to upload his / her personal information. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

[0216] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical application, or improvement of technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A network training method, characterized by, The method comprises: determining an image processing loss of an initial neural network based on support set images and query set images, wherein the initial neural network comprises a support set self-supervised branch and a query set self-supervised branch, the support set self-supervised branch is used to perform a self-supervised training task based on the support set images, and the query set self-supervised branch is used to perform a self-supervised training task based on the query set images; determining self-supervised training data based on the support set images and / or the query set images; determining a self-supervised loss of the initial neural network based on the self-supervised training data, wherein the self-supervised loss comprises a support set self-supervised loss and a query set self-supervised loss; training the initial neural network based on the image processing loss and the self-supervised loss to obtain a target neural network; wherein the target neural network is used to perform an image processing operation on a to-be-processed image to obtain a target image processing result.

2. The method of claim 1, wherein, The method comprises: determining a predicted image processing result of the query set images based on the support set images and the initial neural network; determining the image processing loss based on the predicted image processing result of the query set images and preset labels of the query set images.

3. The method according to claim 1 or 2, characterized in that, The self-supervised loss comprises a support set rotation angle prediction loss. The method comprises: rotating the support set images, and determining the rotated support set images as the self-supervised training data; The method comprises: determining a predicted rotation angle of the rotated support set images by using the initial neural network, and determining the support set rotation angle prediction loss based on the predicted rotation angle of the rotated support set images.

4. The method according to claim 1 or 2, characterized in that, The self-supervised loss comprises a query set rotation angle prediction loss. The method comprises: rotating the query set images, and determining the rotated query set images as the self-supervised training data; The method comprises: determining a predicted rotation angle of the rotated query set images by using the initial neural network, and determining the query set rotation angle prediction loss based on the predicted rotation angle of the rotated query set images.

5. The method according to claim 1 or 2, characterized in that, The self-supervised loss comprises a support set relative position prediction loss. The method comprises: performing image block segmentation on the support set images, and determining a plurality of support set image blocks obtained by segmentation as the self-supervised training data; The method comprises: determining a predicted relative position between any two support set image blocks by using the initial neural network, and determining the support set relative position prediction loss based on the predicted relative position between the any two support set image blocks.

6. The method of claim 1 or 2, wherein, The self-supervised loss comprises a query set relative position prediction loss; The self-supervised training data is determined based on the query set images, comprising: The query set images are divided into image blocks, and the plurality of query set image blocks obtained by the division are determined as the self-supervised training data; The self-supervised loss of the initial neural network is determined based on the self-supervised training data, comprising: The predicted relative positions between any two query set image blocks are determined by using the initial neural network, and the query set relative position prediction loss is determined based on the predicted relative positions between the any two query set image blocks.

7. The method according to claim 1 or 2, characterized in that, The self-supervised loss comprises a support set feature consistency loss; The self-supervised training data is determined based on the support set images, comprising: The support set images are subjected to two different image enhancement processing operations, and the two enhanced support set images obtained are determined as the self-supervised training data; The self-supervised loss of the initial neural network is determined based on the self-supervised training data, comprising: The image features corresponding to each enhanced support set image are respectively determined by using the initial neural network, and the support set feature consistency loss is determined based on the image features corresponding to the two enhanced support set images.

8. The method of claim 1 or 2, wherein, The self-supervised loss comprises a query set feature consistency loss; The self-supervised training data is determined based on the query set images, comprising: The query set images are subjected to two different image enhancement processing operations, and the two enhanced query set images obtained are determined as the self-supervised training data; The self-supervised loss of the initial neural network is determined based on the self-supervised training data, comprising: The image features corresponding to each enhanced query set image are respectively determined by using the initial neural network, and the query set feature consistency loss is determined based on the image features corresponding to the two enhanced query set images.

9. The method of claim 1 or 2, wherein, The initial neural network is trained based on the image processing loss and the self-supervised loss to obtain a target neural network, comprising: The image processing loss and the self-supervised loss are weighted and summed to obtain a total loss; The network parameters of the initial neural network are adjusted based on the total loss to obtain the target neural network.

10. An image processing method characterized by, Comprise: An image to be processed is acquired; An image processing operation is performed on the image to be processed by using a target neural network to obtain a target image processing result; The target neural network is trained based on the method of any one of claims 1 to 9.

11. A network training apparatus, characterized by comprising: Comprise: An image processing loss determination module is configured to determine an image processing loss of an initial neural network based on support set images and query set images, wherein the initial neural network comprises a support set self-supervised branch and a query set self-supervised branch, the support set self-supervised branch is configured to perform a self-supervised training task based on the support set images, and the query set self-supervised branch is configured to perform a self-supervised training task based on the query set images; A self-supervised training data determination module is configured to determine self-supervised training data based on the support set images and / or the query set images; The self-supervised loss determination module is configured to determine a self-supervised loss of the initial neural network based on the self-supervised training data, where the self-supervised loss includes a support set self-supervised loss and a query set self-supervised loss. The training module is configured to train the initial neural network based on the image processing loss and the self-supervised loss to obtain a target neural network. The target neural network is configured to perform an image processing operation on a to-be-processed image to obtain a target image processing result.

12. An image processing apparatus characterized by comprising: The method comprises the following steps: An acquisition module is configured to acquire a to-be-processed image. An image processing module is configured to perform an image processing operation on the to-be-processed image by using the target neural network to obtain a target image processing result. The target neural network is trained based on the method in any one of claims 1 to 9.

13. An electronic device, comprising: The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 10. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 10. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 10. ​ 14. A computer-readable storage medium having stored thereon computer program instructions, wherein, ​

Images