Method for augmenting image feature samples and constructing neural networks

Abstract

The present disclosure provides a method for expanding image feature samples and a method for constructing a neural network for image processing. The method for expanding image feature samples comprises: determining a plurality of auxiliary images corresponding to a plurality of target images, and extracting a plurality of target feature samples and a plurality of corresponding auxiliary feature samples therefrom; for a given auxiliary feature sample: selecting a target feature sample corresponding to the given auxiliary feature sample to form a positive sample pair; and selecting target feature samples to form a plurality of negative sample pairs based on the similarity between the corresponding auxiliary feature sample of each candidate target feature sample and the given auxiliary feature sample, which are used to train a neural network for extracting image features. According to the method for expanding image feature samples, the originally single target image feature sample is expanded by introducing auxiliary image data, and the construction of negative sample pairs is guided based on the sample feature similarity rule, overcoming the scene similarity problem.

Description

Technical Field

[0001] This disclosure relates to image processing, and more specifically, to methods, apparatus, and computer-readable storage media for augmenting image feature samples and constructing neural networks for image processing. Background Technology

[0002] In recent years, much of the latest advancement in fields such as computer perception, speech recognition, and natural language processing has revolved around supervised deep learning, which requires a large amount of manually labeled training data so that machine learning models can learn based on predictions and these labels. For example, in image processing, the ImageNet dataset, with approximately one million labeled images, can be used to train machine learning models to solve problems such as semantic segmentation and object detection. However, in some vertical domains, the amount of manually labeled training data may be limited, or machine learning models trained on a given amount of labeled data may not be able to effectively address these issues. Therefore, unsupervised learning has been widely applied in domains where labeled data is scarce.

[0003] On the other hand, although there is some research on unsupervised learning, most methods rely on one type of data to learn model parameters. When the differences between samples of this type of data are not significant, it makes it difficult to learn a good machine learning model.

[0004] Therefore, there is a need for an improved method for constructing neural networks and a corresponding method for providing image data training samples in scenarios where the differences between image data samples may not be significant. Summary of the Invention

[0005] According to one aspect of this disclosure, a method for expanding image feature samples is provided, comprising: determining a plurality of corresponding auxiliary images for a plurality of target images, and extracting a plurality of target feature samples and a plurality of corresponding auxiliary feature samples from the plurality of target images and the plurality of corresponding auxiliary images; for a given auxiliary feature sample among the plurality of auxiliary feature samples, selecting feature samples from the plurality of target feature samples to form a positive sample pair and a plurality of negative sample pairs, wherein: a target feature sample corresponding to the given auxiliary feature sample is selected from the plurality of target feature samples to form the positive sample pair; and from a plurality of candidate target feature samples that do not correspond to the given auxiliary feature sample among the plurality of target feature samples, a plurality of target feature samples are selected to form the plurality of negative sample pairs based on the similarity between the corresponding auxiliary feature sample of each candidate target feature sample and the given auxiliary feature sample, wherein the positive sample pair and the plurality of negative sample pairs are used to train a neural network for extracting image features.

[0006] According to another aspect of this disclosure, an apparatus for expanding image feature samples is provided, comprising: a processor; and a memory storing computer program instructions, wherein, when the computer program instructions are executed by the processor, the processor performs the following steps: determining a plurality of corresponding auxiliary images for a plurality of target images, and extracting a plurality of target feature samples and a plurality of corresponding auxiliary feature samples from the plurality of target images and the plurality of corresponding auxiliary images; for a given auxiliary feature sample among the plurality of auxiliary feature samples, selecting feature samples from the plurality of target feature samples to form a positive sample pair and a plurality of negative sample pairs, wherein: a target feature sample corresponding to the given auxiliary feature sample is selected from the plurality of target feature samples to form the positive sample pair, and a plurality of target feature samples not corresponding to the given auxiliary feature sample are selected from the plurality of target feature samples to form the plurality of negative sample pairs based on the similarity between the corresponding auxiliary feature sample of each candidate target feature sample and the given auxiliary feature sample, wherein the positive sample pair and the plurality of negative sample pairs are used to train a neural network for extracting image features.

[0007] According to another aspect of this disclosure, an apparatus for expanding image feature samples is provided, comprising: a feature sample extraction unit configured to determine a plurality of corresponding auxiliary images for a plurality of target images, and extract a plurality of target feature samples and a plurality of corresponding auxiliary feature samples from the plurality of target images and the plurality of corresponding auxiliary images; and a sample pair construction unit configured to, for a given auxiliary feature sample among the plurality of auxiliary feature samples, select feature samples from the plurality of target feature samples to form a positive sample pair and a plurality of negative sample pairs, wherein: the sample pair construction unit selects target feature samples corresponding to the given auxiliary feature sample from the plurality of target feature samples to form the positive sample pair, and the sample pair construction unit selects a plurality of target feature samples from a plurality of candidate target feature samples that do not correspond to the given auxiliary feature sample to form the plurality of negative sample pairs based on the similarity between the corresponding auxiliary feature samples of each candidate target feature sample and the given auxiliary feature sample, wherein the positive sample pair and the plurality of negative sample pairs are used to train a neural network for extracting image features.

[0008] According to another aspect of this disclosure, a method for constructing a neural network for image processing is provided, comprising: constructing a neural network including a target encoder and an auxiliary encoder; configuring the target encoder and the auxiliary encoder to extract multiple target feature samples and multiple corresponding auxiliary feature samples from multiple target images and multiple corresponding auxiliary images, respectively; for a given auxiliary feature sample among the multiple auxiliary feature samples, selecting feature samples from the multiple target feature samples to form a positive sample pair and multiple negative sample pairs, wherein: the positive sample pair is formed by selecting target feature samples corresponding to the given auxiliary feature sample from the multiple target feature samples; and multiple target feature samples are selected from multiple candidate target feature samples that do not correspond to the given auxiliary feature sample from the multiple target feature samples based on the similarity between each candidate target feature sample and the given auxiliary feature sample. The target feature samples constitute the plurality of negative sample pairs; for a given target feature sample among the plurality of target feature samples, feature samples are selected from the plurality of auxiliary feature samples to form an additional positive sample pair and a plurality of additional negative sample pairs, wherein: an auxiliary feature sample corresponding to the given target feature sample is selected from the plurality of auxiliary feature samples to form an additional positive sample pair, and a plurality of auxiliary feature samples are selected from a plurality of candidate auxiliary feature samples that do not correspond to the given target feature sample to form a plurality of additional negative sample pairs based on the similarity between each candidate auxiliary feature sample and the given target feature sample; the loss of the neural network is calculated based on the positive sample pairs and the plurality of negative sample pairs, the additional positive sample pairs and the plurality of additional negative sample pairs; and the parameters of the target encoder and the auxiliary encoder are updated based on the loss.

[0009] According to another aspect of this disclosure, an apparatus for constructing a neural network for image processing is provided, comprising: a processor; and a memory storing computer program instructions, wherein, when the computer program instructions are executed by the processor, the processor performs the following steps: constructing a neural network including a target encoder and an auxiliary encoder; configuring the target encoder and the auxiliary encoder to extract a plurality of target feature samples and a plurality of corresponding auxiliary feature samples from a plurality of target images and a plurality of corresponding auxiliary images, respectively; for a given auxiliary feature sample among the plurality of auxiliary feature samples, selecting feature samples from the plurality of target feature samples to form a positive sample pair and a plurality of negative sample pairs, wherein: the positive sample pair is formed by selecting a target feature sample corresponding to the given auxiliary feature sample from the plurality of target feature samples; and the negative sample pair is formed by selecting a target feature sample from the plurality of target feature samples that does not correspond to the given auxiliary feature sample, based on each candidate target feature sample... The similarity between the target feature sample and the given auxiliary feature sample is used to select multiple target feature samples to form multiple negative sample pairs; for a given target feature sample among the multiple target feature samples, feature samples are selected from the multiple auxiliary feature samples to form an additional positive sample pair and multiple additional negative sample pairs, wherein: an auxiliary feature sample corresponding to the given target feature sample is selected from the multiple auxiliary feature samples to form an additional positive sample pair, and multiple auxiliary feature samples not corresponding to the given target feature sample are selected from the multiple auxiliary feature samples to form multiple additional negative sample pairs based on the similarity between each candidate auxiliary feature sample and the given target feature sample; the loss of the neural network is calculated based on the positive sample pair and the multiple negative sample pairs, the additional positive sample pair and the multiple additional negative sample pairs; and the parameters of the target encoder and the auxiliary encoder are updated based on the loss.

[0010] According to another aspect of this disclosure, an apparatus for constructing a neural network for image processing is provided, wherein the neural network includes a target encoder and an auxiliary encoder. The construction apparatus includes: a feature sample extraction unit configured to configure the target encoder and the auxiliary encoder to extract multiple target feature samples and multiple corresponding auxiliary feature samples from multiple target images and multiple corresponding auxiliary images, respectively; and a sample pair construction unit configured to, for a given auxiliary feature sample among the multiple auxiliary feature samples, select feature samples from the multiple target feature samples to form a positive sample pair and multiple negative sample pairs, wherein: the positive sample pair is formed by selecting target feature samples corresponding to the given auxiliary feature sample from the multiple target feature samples; and the multiple negative sample pairs are formed by selecting multiple target feature samples from the multiple target feature samples that do not correspond to the given auxiliary feature sample, based on the similarity between each candidate target feature sample and the given auxiliary feature sample; for the multiple target... For a given target feature sample in the feature samples, feature samples are selected from the plurality of auxiliary feature samples to form an additional positive sample pair and a plurality of additional negative sample pairs, wherein: an auxiliary feature sample corresponding to the given target feature sample is selected from the plurality of auxiliary feature samples to form an additional positive sample pair, and a plurality of auxiliary feature samples are selected from a plurality of candidate auxiliary feature samples that do not correspond to the given target feature sample to form a plurality of additional negative sample pairs based on the similarity between each candidate auxiliary feature sample and the given target feature sample; a loss calculation unit is configured to calculate the loss of the neural network based on the positive sample pair and the plurality of negative sample pairs, the additional positive sample pair and the plurality of additional negative sample pairs; and a neural network update unit is configured to update the parameters of the target encoder and the auxiliary encoder based on the loss.

[0011] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed, implement the above-described method for expanding image feature samples and / or the method for constructing a neural network for image processing. Attached Figure Description

[0012] These and / or other aspects and advantages of this disclosure will become clearer and more readily understood from the following detailed description of embodiments of this disclosure taken in conjunction with the accompanying drawings, wherein:

[0013] Figure 1 A flowchart of a method for expanding image feature samples according to an embodiment of the present disclosure is shown.

[0014] Figure 2 A schematic diagram illustrating an example of an expanded image feature sample according to an embodiment of the present disclosure is shown.

[0015] Figure 3 A schematic diagram of another example of an expanded image feature sample according to an embodiment of the present disclosure is shown.

[0016] Figure 4 A schematic diagram illustrating yet another example of an expanded image feature sample according to an embodiment of the present disclosure is shown.

[0017] Figure 5A and Figure 5B A schematic diagram illustrating an example of the negative sample pair construction process when expanding image feature samples according to an embodiment of the present disclosure is shown.

[0018] Figure 6 A flowchart illustrating a method for constructing a neural network for image processing according to an embodiment of the present disclosure is shown.

[0019] Figure 7 A schematic diagram illustrating an example of the construction of a neural network for image processing according to an embodiment of the present disclosure is shown.

[0020] Figure 8 A schematic diagram of another example of the construction of a neural network for image processing according to embodiments of the present disclosure is shown.

[0021] Figure 9 A schematic diagram illustrating examples of unsupervised learning modules and supervised learning modules in a neural network for image processing according to embodiments of the present disclosure is shown.

[0022] Figure 10 A schematic diagram of another example of a supervised learning module in a neural network for image processing according to an embodiment of the present disclosure is shown.

[0023] Figure 11 A schematic hardware block diagram of an apparatus for expanding image feature samples according to an embodiment of the present disclosure is shown.

[0024] Figure 12 A schematic structural block diagram of an apparatus for expanding image feature samples according to an embodiment of the present disclosure is shown.

[0025] Figure 13 A schematic hardware block diagram of an apparatus for constructing a neural network for image processing, according to an embodiment of the present disclosure, is shown.

[0026] Figure 14 A schematic structural block diagram of an apparatus for constructing a neural network for image processing, according to an embodiment of the present disclosure, is shown. Detailed Implementation

[0027] To enable those skilled in the art to better understand this disclosure, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] First, a brief overview of the basic ideas behind the improved neural network construction and the corresponding image feature training sample expansion technique disclosed in this disclosure is provided. As mentioned earlier, most unsupervised learning methods currently rely on a single type of data to learn model parameters, which is challenging when the differences between data samples of that type are not significant. As an illustrative example in the field of image processing, product defect detection is a common scenario where machine learning models can be applied to detection tasks, thus playing a crucial role in product quality control. For instance, consider the scenario of defect detection using infrared images of objects such as solar panels and railway tracks. In this case, machine learning can be performed using infrared image feature training samples extracted from the infrared images, and the resulting machine learning model can be used to detect and locate defects in subsequent infrared images of the object under test. However, because infrared images contain relatively little information, and the differences between different samples are not significant, learning based solely on infrared image feature samples results in too small a difference between normal and abnormal infrared image feature samples, making model learning difficult. In summary, in scenarios where the differences between data samples may not be significant, existing machine learning methods may struggle to learn or achieve good learning results, failing to meet the growing demands in the field of machine learning.

[0029] In view of this, this disclosure proposes the idea of ​​introducing different data sources for multi-view comparative learning, thereby improving the learning of model parameters for extracting target image features based on target image data with the help of auxiliary image data. The inventors of this disclosure have noted that for the same scene (e.g., the same photovoltaic panel), different types of image data can be used to capture image information of that scene, obtaining image data of different types or sources (or "different perspectives") for the same scene. For example, visible light images, infrared images, depth images, and semantic segmentation images can be obtained for the same scene. Based on obtaining multi-view image data for the same scene, even if the differences between different image feature samples of a certain type of image (e.g., infrared image) are not significant, making it difficult to learn the parameters of the infrared image feature extraction model, another type of image can still be introduced to assist in the training of the infrared image feature extraction model, resulting in better learning performance. In other words, while image data of a certain type or viewpoint may not be sufficient for good model training, specific properties of a particular object in the scene (e.g., a defective solar panel) can be shared by image data from multiple viewpoints obtained for that scene, and can be viewed as mutual information shared between different viewpoints. Therefore, by maximizing the mutual information between different viewpoints of the same scene as the goal of machine learning, the model can learn cross-viewpoint invariant features, thus enabling it to perform well in real-world applications.

[0030] To perform multi-view contrastive learning, it is desirable to map semantically adjacent variables more closely within the embedding space while pushing away irrelevant variables. This requires constructing corresponding positive and negative sample pairs to input into the neural network model for the learning process. It can be understood that a positive sample pair in this disclosure consists of two samples from different image data sources corresponding to the same scene. In other words, the images of these two samples have overlapping areas or were acquired at the same time (e.g., they may contain the same detection object, such as the same solar panel). Therefore, the scenes corresponding to these two samples are the same, and they share some image information or cross-view invariant information. A negative sample pair, on the other hand, consists of two samples from different data sources corresponding to different scenes. In other words, the images of these two samples were taken for different scenes, so theoretically, there should be some differences between them. Currently, although there are some methods for constructing image feature training samples, most use image enhancement methods to generate positive samples, while other images are negative samples. Obviously, this sample construction method is not suitable for training models for multi-view contrastive learning. Therefore, the following will first introduce the image feature sample augmentation method proposed in this disclosure, and on this basis, introduce the improved construction and training method of the neural network based on multi-view contrast learning proposed in this disclosure.

[0031] It should be understood that, for ease of explanation, the target image (or target feature sample) is used as an infrared image (or infrared feature sample), and the auxiliary image (or auxiliary feature sample) is used as a visible light image (or visible light feature sample) as examples to describe the image feature sample augmentation technique and the improved neural network construction technique proposed in this disclosure. However, it is understood that "target" and "auxiliary" discussed in this disclosure are relative concepts. Specifically, the "target" image or feature refers to the type of image that the neural network trained in this disclosure is interested in during actual subsequent use, while the "auxiliary" image or feature refers to image data from another source that assists in the training process of the encoder extracting target image features. For example, in a defect detection scenario based on infrared images, the target image can be an infrared image, while the auxiliary image can be another source of data that assists in the training process of the encoder extracting infrared features. It is understood that the two can be interchanged; that is, the target image can also be a visible light image, while the auxiliary image is an infrared image that plays an auxiliary role. Furthermore, the image feature sample augmentation and the construction and training of neural networks based on multi-view contrast learning disclosed herein can also utilize images from other sources, such as depth images, semantic segmentation images, etc. In other words, this disclosure can use any of the following image types as the target image: infrared images, visible light images, depth images, semantic segmentation images, and others, while using any other image type as an auxiliary image; this disclosure is not limited thereto.

[0032] Furthermore, the subsequent use cases (or downstream tasks) of the neural network disclosed herein can be various practical application scenarios based on target image data, such as semantic segmentation, object detection, etc., as described above, for example, infrared image defect detection of photovoltaic panels, but this disclosure is not limited thereto. In this disclosure, by using auxiliary image data to augment the target image feature samples, additional information is provided for the network to better learn the parameters, thereby enabling multi-view comparative learning based on the augmented image feature training samples, allowing the machine learning model to achieve better results in downstream tasks.

[0033] Method for augmenting image feature samples

[0034] Figure 1 A flowchart of a method for expanding image feature samples according to an embodiment of the present disclosure is shown. Figure 2 A schematic diagram illustrating an example of expanding image feature samples according to an embodiment of the present disclosure is shown. The following details the process in conjunction with... Figure 1 and Figure 2 Describe the method used to augment image feature samples.

[0035] like Figure 1As shown, in step S101, multiple auxiliary images are determined for multiple target images, and multiple target feature samples and multiple auxiliary feature samples are extracted from the multiple target images and the corresponding multiple auxiliary images.

[0036] As an illustrative example, the target image can be an infrared image, while the auxiliary image can be a visible light image. Furthermore, for any one of the multiple infrared images, there exists a visible light image among the multiple visible light images that captures the same scene as the infrared image. For example, combining... Figure 2 As shown, assuming there are N infrared images and N corresponding visible light images, the first infrared image and the first visible light image can be captured or obtained for the same first scene, while the second infrared image and the second visible light image can be captured or obtained for the same second scene, and so on. Therefore, based on this correspondence, multiple auxiliary images can be determined for multiple target images. It is understood that the infrared images and visible light images can be captured using any suitable image capturing device, or obtained through certain image processing. This disclosure does not limit the method of capturing or obtaining infrared images and visible light images, as long as the above correspondence is satisfied.

[0037] Additionally, in step S101, an encoder can be used to extract features from the target image and the corresponding auxiliary image to obtain target feature samples and corresponding auxiliary feature samples. As an illustrative example, this encoder can be part of a deep convolutional neural network. For instance, an encoder can be a general term for the feature extraction layers of a neural network, typically including convolutional layers, pooling layers, activation functions, etc. The encoder can be used to extract compact information from an image that characterizes the image information for subsequent tasks such as semantic segmentation and object detection. For example, an encoder used to extract features from infrared images can be applied to a defect detection task based on infrared images. For instance, for the commonly used ResNet network, the remaining network layers after removing the fully connected correlation layers can serve as an encoder.

[0038] Continue to combine Figure 2 The description can utilize the auxiliary feature encoder f α Extract auxiliary feature samples y from N auxiliary images respectively. op,1 ~y op,N This serves as an auxiliary feature sample set. Similarly, the target feature encoder f can be used. β Extract target feature samples y from N target images respectively. in,1 ~y in,N This serves as the target feature sample set. Based on this, a sample can be selected from both the auxiliary feature sample set and the target feature sample set to construct positive and negative sample pairs for multi-view contrastive learning.

[0039] return Figure 1 In step S102, for a given auxiliary feature sample among the plurality of auxiliary feature samples, feature samples are selected from the plurality of target feature samples to form a positive sample pair and multiple negative sample pairs. Subsequently, the positive sample pair and multiple negative sample pairs can be used to train a neural network for extracting image features, for example, based on the idea of ​​multi-view contrastive learning, to train a neural network for extracting target image features. As discussed above, for contrastive learning, it is necessary to construct positive sample pairs to include two image feature samples for the same scene, and to construct negative sample pairs to include two image feature samples for different scenes. Furthermore, for multi-view learning, image feature samples can be extracted from image data from different sources. The following continues in conjunction with... Figure 1 and Figure 2 This paper describes how to select samples from multiple target feature samples to form sample pairs, given auxiliary feature samples, from the perspectives of constructing positive sample pairs and constructing negative sample pairs.

[0040] On the one hand, selecting feature samples from multiple target feature samples to form a positive sample pair can include: selecting target feature samples from multiple target feature samples that correspond to a given auxiliary feature sample to form the positive sample pair. For example, combining... Figure 2 As shown, for a given value i from 1 to N, the image feature sample y op,i and y in,i These two are mutually corresponding, or rather, they were captured or acquired from the same scene, therefore they can be considered as a positive sample pair. In other words, for multiple auxiliary feature samples y within the auxiliary feature sample set... op,1 ~y op,N Given auxiliary feature sample y op,i It can be derived from multiple target feature samples y within the target feature sample set. in,1 ~y in,N Select the corresponding target feature sample y in,i , thus making y op,i With y in,i This forms a positive sample pair. It can be understood that for multiple auxiliary feature samples y... op,1 ~y op,N Other given auxiliary feature samples in the dataset can be similarly derived from multiple target feature samples y. in,1 ~y in,N The selection of corresponding target feature samples to form more positive sample pairs is not elaborated here.

[0041] On the other hand, selecting feature samples from multiple target feature samples to form multiple negative sample pairs may include: selecting multiple target feature samples from multiple candidate target feature samples that do not correspond to the given auxiliary feature sample to form the multiple negative sample pairs. Continuing with this... Figure 2 Description: For multiple auxiliary feature samples y in the auxiliary feature sample set... op,1 ~y op,N Given auxiliary feature sample y op,i We can first select from multiple target feature samples y within the target feature sample set. in,1 ~y in,N Exclude target feature samples y that already form positive sample pairs in,i That is, excluding y in,i The remaining target feature samples, excluding those mentioned above, are considered as multiple candidate target feature samples, from which target feature samples are selected to form multiple negative sample pairs. For example, y can be selected from multiple candidate target feature samples. in,2 y in,j and y in,N , thus making y op,i With y in,2 y op,i With y in,j y op,i With y in,N This forms three negative sample pairs.

[0042] Multiple negative sample pairs can be formed by selecting multiple target feature samples from multiple candidate target feature samples according to a predetermined feature sample selection rule. The inventors recognized that in some domains, the symmetry or similarity of images can make learning a good network difficult. To enable multi-view contrast learning to be effective, the two feature samples constituting a negative sample pair should have low similarity to overcome the sample similarity problem. To illustrate the sample similarity problem in the learning process, the following example is provided. For instance, for scenes m and n that may be very similar to each other, infrared images m and n, and visible light images m and n can be obtained by taking pictures or performing image processing, respectively, and visible light image feature samples y can be extracted accordingly. op,m and visible light image feature samples y op,n and the corresponding infrared image feature sample y in,m and infrared image feature samples y in,n If no rules are imposed on the construction process of negative sample pairs, then for a given visible light image feature sample y... op,m During the construction of negative sample pairs, it is highly likely that infrared image feature samples will be selected from multiple candidate infrared feature samples. in,n , so as to make y op,m With yin,n As negative sample pairs, they are used for machine learning; however, since scene m and scene n are quite similar, the image feature samples y extracted from them... op,m With y in,n There is also a certain degree of similarity between the images. Using image feature samples with similar scenes as negative sample pairs for learning will make it difficult for the learning network.

[0043] This disclosure considers that, to avoid pairing two samples with similar scenes into negative sample pairs, certain rules need to be applied to the construction process of negative sample pairs. Therefore, to reduce the adverse effects of similar scenes on machine learning, this disclosure applies certain rules to the construction process of negative sample pairs based on the weighted sampling idea of ​​sample similarity, thereby constructing suitable negative sample pairs for machine learning. In this disclosure, the scene similarity between two image feature samples to be selected can be evaluated based on the similarity between each candidate target feature sample and a given auxiliary feature sample. Based on the calculated similarity, multiple target feature samples are selected from multiple candidate target feature samples to form multiple negative sample pairs, thereby avoiding the construction of negative sample pairs between two samples with similar scenes.

[0044] One approach is to directly utilize the similarity between each candidate target feature sample and a given auxiliary feature sample; that is, to directly calculate the candidate target feature sample y. in,1 ~y in,N (where y is excluded) in,i ) and given auxiliary feature sample y op,i The similarity between them is used to evaluate the feature samples y of each candidate target. in,1 ~y in,N (where y is excluded) in,i The corresponding scenarios and the given auxiliary feature samples y op,i The similarity between corresponding scenes is used to avoid selecting samples with similar scenes to form negative sample pairs. The similarity calculation between image feature samples can be performed using methods such as cosine similarity, Euclidean distance, and Hamming distance; this disclosure does not impose any limitations on this method.

[0045] As another approach, since the information contained in the target image may be relatively limited, directly calculating the similarity between target image feature samples and auxiliary image feature samples may not be very accurate in assessing the similarity between scenes. Therefore, the similarity calculation between target image feature samples and auxiliary image feature samples can be transformed into a similarity calculation between auxiliary image feature samples of the same type. This allows for a more accurate reflection of the similarity between the scenes corresponding to the two image feature samples being evaluated, under the global guidance of the auxiliary image data, thus better utilizing auxiliary image features to guide the sampling of target image feature samples. The weighted sampling idea for target image feature samples proposed in this disclosure overcomes the problem of samples from similar scenes being selected to form negative sample pairs. Furthermore, in scenarios where the differences between scenes in the target images are not significant and the similarity is high, using auxiliary feature samples to guide the selection of target feature samples allows the network to learn subtle differences in the target image. For example, selecting feature samples from multiple target feature samples to form multiple negative sample pairs may include: selecting multiple target feature samples from multiple candidate target feature samples that do not correspond to the given auxiliary feature sample, based on the similarity between the corresponding auxiliary feature sample of each candidate target feature sample and the given auxiliary feature sample, to form the multiple negative sample pairs. Specifically, for a candidate target feature sample, its corresponding auxiliary feature sample can be found, then the similarity between the corresponding auxiliary feature sample and the given auxiliary feature can be calculated, and finally the weight of selecting the candidate target sample is determined based on the calculated similarity. Figure 2 Description, for a certain candidate target feature sample y in,j It is possible to find its corresponding auxiliary feature sample y. op,j Then calculate the corresponding auxiliary feature sample y. op,j With a given auxiliary feature y op,i The similarity between the samples is used to determine the candidate target sample y. in,j The selected weights are used to avoid selecting feature samples with similar scenes to form negative sample pairs. In other words, when the similarity calculated for two image feature samples to be evaluated is high, it means that although these two samples were taken or acquired in different scenes, the similarity between the corresponding two scenes is high, and therefore they are not suitable as negative sample pairs for learning, thus assigning them lower sampling weights. Figure 2 As shown, k target feature samples can be selected from the candidate feature samples to form k negative sample pairs. It can be understood that the value of k can be set empirically or adjusted through comparative experiments. For example, k target feature samples can be selected in ascending order of similarity to form k negative sample pairs.

[0046] Finally, given the auxiliary feature samples, a positive sample pair and k negative sample pairs can be used for the learning and training of a neural network to extract image features. This can be understood as follows: Figure 2 The diagram only illustrates one positive sample pair and k negative sample pairs generated given a specific i-th auxiliary feature sample. However, more positive and negative sample pairs can be constructed in a similar manner for each of the N auxiliary feature samples, and these more positive and negative sample pairs constructed given the auxiliary feature samples can be input into the neural network for learning.

[0047] The above describes the construction of positive and negative sample pairs using auxiliary feature samples from the auxiliary feature sample set and selecting image feature samples from the target feature sample set. That is, constructing positive and negative sample pairs with auxiliary features as references. Similarly, it can be understood that target feature samples can be given from the target feature sample set, and image feature samples can be selected from the auxiliary feature sample set to construct positive and negative sample pairs. That is, constructing positive and negative sample pairs with target features as references. The following section combines... Figure 3 This paper describes how to select samples from multiple auxiliary feature samples to form sample pairs, given a target feature sample, from the perspectives of constructing positive sample pairs and constructing negative sample pairs.

[0048] like Figure 3 As shown, on the one hand, for a given target feature sample among the plurality of target feature samples, selecting feature samples from the plurality of auxiliary feature samples to form an additional positive sample pair includes: selecting auxiliary feature samples corresponding to the given target feature sample from the plurality of auxiliary feature samples to form an additional positive sample pair. For example, combined with Figure 3 As shown, and similar to the above combination Figure 2 The discussion concerns multiple target feature samples y in the target feature sample set. in,1 ~y in,N Given auxiliary feature sample y in,i It can be derived from multiple auxiliary feature samples y within the auxiliary feature sample set. op,1 ~y op,N Select the corresponding auxiliary feature sample y op,i , thus making y in,i With y op,i This forms a positive sample pair. It can be understood that for multiple target feature samples y... in,1 ~y in,N Other given target feature samples in the dataset can be similarly derived from multiple auxiliary feature samples y. op,1 ~y op,NThe selection of corresponding target feature samples to form more positive sample pairs is not elaborated here.

[0049] On the other hand, selecting feature samples from multiple auxiliary feature samples to form multiple additional negative sample pairs includes: selecting multiple auxiliary feature samples from multiple candidate auxiliary feature samples that do not correspond to the given target feature sample to form multiple additional negative sample pairs. Continuing with... Figure 3 Description, for multiple target feature samples y in,1 ~y in,N Given target feature sample y in,i We can first start with multiple auxiliary feature samples y from the auxiliary feature sample set. op,1 ~y op,N Auxiliary feature samples y that already form positive sample pairs are excluded. op,i That is, excluding y op,i The remaining auxiliary feature samples are used as multiple candidate auxiliary feature samples, from which feature samples are selected to form multiple additional negative sample pairs.

[0050] Combined with the above Figure 2 The idea of ​​weighted sampling based on sample similarity described is similar. Figure 3 In the process of selecting samples from the auxiliary feature sample set to construct negative sample pairs, sample similarity can also be used to avoid feature samples with similar scenes being constructed as negative sample pairs. In this disclosure, the scene similarity between two image feature samples to be selected can be evaluated based on the similarity between each candidate auxiliary feature sample and a given target feature sample. Based on the calculated similarity, multiple target feature samples are selected from multiple candidate target feature samples to form multiple negative sample pairs, thereby avoiding two samples with similar scenes being constructed as negative sample pairs.

[0051] One approach is to directly utilize the similarity between each candidate auxiliary feature sample and the given target feature sample; that is, to directly calculate the similarity between the candidate auxiliary feature samples y. op,1 ~y op,N (excluding y) op,i ) and given target feature sample y in,i The feature distance between them is used to evaluate each candidate auxiliary feature sample y. op,1 ~y op,N (excluding y) op,i The corresponding scenarios and the given target feature samples y in,i The similarity between the corresponding scenes is used to avoid selecting samples with similar scenes to form negative sample pairs.

[0052] As one implementation approach, the similarity calculation between target image feature samples and auxiliary image feature samples can be transformed into a similarity calculation between auxiliary image feature samples of the same type, allowing sampling to be performed under the global guidance of the auxiliary image data. For example, given a target feature sample, its corresponding auxiliary feature sample can be found, then the similarity between each candidate auxiliary feature sample and its corresponding auxiliary feature sample can be calculated, and finally, the weight for selecting the candidate auxiliary sample is determined based on the calculated similarity. Figure 3 Description, for a given target feature sample y in,i It is possible to find its corresponding auxiliary feature sample y. opi Then calculate a candidate auxiliary sample y op,j The corresponding auxiliary feature sample y opi i The similarity between the samples is used to determine the candidate auxiliary sample y. op,j The selected weights are used to prevent feature samples with similar scenes from being selected to form negative sample pairs. For example... Figure 3 As shown, k auxiliary feature samples can be selected to form k additional negative sample pairs, and they can be selected in order of similarity from low to high.

[0053] Finally, in addition to combining the above Figure 2 The described sample pairs, formed given auxiliary feature samples, are used for the learning and training of a neural network to extract image features. This can also be used to... Figure 3 The additional sample pairs shown are used for learning and training the neural network given the target feature samples. It can be understood that... Figure 3 The diagram only illustrates one additional positive sample pair and k additional negative sample pairs generated given a specific i-th target feature sample. However, for each of the N target feature samples, more additional positive and negative sample pairs can be generated in a similar manner and fed into the neural network for learning.

[0054] It is understandable that, in order to construct image feature training samples suitable for multi-view comparison learning, one could, as follows: Figure 2 As shown, using auxiliary feature samples as a reference, samples are drawn from the target feature sample set to form positive and negative sample pairs; and it can be done as follows: Figure 3 As shown, using the target feature samples as a reference, samples are drawn from the auxiliary feature sample set to form additional positive and negative sample pairs. Combined with... Figure 2 and Figure 3As shown, feature extraction can be performed using a single target feature encoder and a single auxiliary feature encoder. However, it should be noted that the embodiments of this disclosure are not limited to the above-described scenario. For example, in this disclosure, two encoders can be used for image feature extraction for each of the target image and the auxiliary image, thereby combining... Figure 2 and Figure 3 The target feature sample set shown is subdivided into two target feature sample sets, and the auxiliary feature sample set is split into two auxiliary feature sample sets, thereby better constructing positive and negative sample pairs. As an illustrative example, this disclosure can employ the concept of consistency regularization to configure the corresponding feature extraction encoder for better integration with multi-view contrastive learning. For example, the encoder can be configured based on ideas such as Π-Model, Temporal Ensembling, and Mean Teacher to construct positive and negative sample pairs more suitable for multi-view contrastive learning. For completeness of explanation, the following combines... Figure 4 This section describes a sample augmentation method based on the Mean Teacher concept combined with multi-perspective comparative learning.

[0055] like Figure 4 As shown, the process of constructing positive and negative sample pairs based on the first two image feature extraction channels can be compared with... Figure 2 Similar to what is described in [the text], the selection of target feature samples is performed given auxiliary feature samples; the process of constructing positive and negative sample pairs based on the latter two image feature extraction channels can be compared with [the text]. Figure 3 Similar to what is described in [the text], it involves selecting auxiliary feature samples given target feature samples. The difference is that... Figure 4 The example employs two auxiliary feature encoders and two target feature encoders for feature extraction. Accordingly, extracting multiple target feature samples from the multiple target images includes: extracting multiple first target feature samples using a first target encoder, and extracting multiple second target feature samples using a second target encoder, wherein the second target encoder is a weighted average version of the first target encoder. Furthermore, extracting multiple auxiliary feature samples from the multiple auxiliary images includes: extracting multiple first auxiliary feature samples using a first auxiliary encoder, and extracting multiple second auxiliary feature samples using a second auxiliary encoder, wherein the second auxiliary encoder is a weighted average version of the first auxiliary encoder.

[0056] It's understandable that configuring the encoder based on the Mean Teacher concept allows us to use the first and second target encoders as a student and teacher model in one Mean Teacher network, respectively, and the first and second auxiliary encoders as a student and teacher model in another Mean Teacher network. For any Mean Teacher network, the teacher model can be considered an averaged version of the student model's weights; both models have the same network structure but different update methods. Using the Mean Teacher concept to configure the encoder involves obtaining model weights through averaging rather than predicting labels, which is particularly suitable for large datasets and online training, resulting in better sample construction.

[0057] like Figure 4 As shown, the first auxiliary feature encoder f can be used. α Extract auxiliary feature samples y from N auxiliary images respectively. op,1 ~y op,N This serves as the first auxiliary feature sample set; and the second target feature encoder f can be used. β Extract target feature samples y' from N target images respectively. in,1 ~y' in,N This serves as the second target feature sample set. Similarly, the first target feature encoder f can be used. β Extract target feature samples y from N target images respectively. in,1 ~y in,N This serves as the first target feature sample set; and the second auxiliary feature encoder f can be used. α Extract auxiliary feature samples y' from each of the N auxiliary images. op,1 ~y' op,N This serves as the second auxiliary feature sample set. The second auxiliary feature encoder f... α ' is the first auxiliary feature encoder f α The model weight average version, and both satisfy f α '=εf α '+(1-ε)f α , where ε is a weighting parameter between 0 and 1; and the second target feature encoder f β ' is the first target feature encoder f β The model weight average version, and both satisfy f β '=εf β '+(1-ε)f βε is a weighting parameter between 0 and 1. The weighting parameter can be gradually increased as the learning process progresses, thereby continuously refining the learning model to achieve better results. Accordingly, the following will continue to combine... Figure 4 This describes the construction process of corresponding positive and negative sample pairs when configuring each encoder based on the Mean Teacher concept.

[0058] On the one hand, refer to Figure 4 The first two image feature extraction channels shown describe the construction process of positive and negative sample pairs given auxiliary feature samples. Specifically, for a given first auxiliary feature sample among multiple first auxiliary feature samples:

[0059] (1) The process of constructing positive sample pairs may include: selecting second target feature samples from multiple second target feature samples that correspond to a given first auxiliary feature sample to form positive sample pairs. For example, combining Figure 4 As shown, for multiple first auxiliary feature samples y in the first auxiliary feature sample set op,1 ~y op,N Given the first auxiliary feature sample y op,i It can be derived from multiple second target feature samples y' within the second target feature sample set. in,1 ~y' in,N Select the corresponding target feature sample y' in,i , thus making y op,i With y' in,i This constitutes a positive sample pair.

[0060] (2) The process of constructing negative sample pairs may include: selecting multiple second target feature samples from multiple candidate second target feature samples that do not correspond to a given first auxiliary feature sample, based on the similarity between each candidate second target feature sample and the given first auxiliary feature sample, to form multiple negative sample pairs. It can be understood that, to overcome the problem of selecting samples with similar scenes to construct negative sample pairs, the construction of negative sample pairs can also be guided by the same weighted sampling idea based on sample similarity. For example, sampling can be performed based on the direct similarity calculation between each candidate second target feature sample and the given first auxiliary feature sample; or sampling can be performed based on the similarity between the corresponding second auxiliary feature sample of each candidate second target feature sample and the given first auxiliary feature sample. For example, combining... Figure 4 As shown, for multiple auxiliary feature samples y in the first auxiliary feature sample set op,1 ~y op,N Given auxiliary feature sample y op,i We can first select from multiple target feature samples y' within the second target feature sample set. in,1 ~y'in,N Exclude target feature samples y' that already form positive sample pairs in,i These serve as multiple candidate second target feature samples, allowing for sampling based on sample similarity rules.

[0061] The following combination Figure 5A Specifically, this describes the similarity calculation of feature samples given a first auxiliary feature sample, and the process of constructing negative sample pairs based on weighted sampling. Combined with... Figure 5A Description, given a first auxiliary feature sample y op,i In the case of a candidate second target feature sample y' within the second target feature sample set, in,j It is possible to find its corresponding auxiliary feature sample y' in the second auxiliary feature sample set. op,j Then calculate the corresponding auxiliary feature sample y' op,j With a given auxiliary feature y op,i The similarity between the samples is used to determine the candidate target sample y'. in,j Selected weight w i,j This avoids selecting feature samples with similar scenes to construct negative sample pairs.

[0062] For example, based on the calculated sample similarity, y' with relatively low similarity can be selected from multiple candidate target feature samples. in,2 y' in,j and y' in,N , thus making y op,i With y' in,2 y op,i With y' in,j y op,i With y' in,N Three negative sample pairs are constructed. Optionally, instead of selecting from all N candidate target feature samples, this disclosure can set a queue of length C in the second target feature sample set, and the extracted second target feature samples will be cached in the queue of length C. When the queue is full, the old samples will be deleted. Accordingly, in the process of constructing negative sample pairs, k negative samples can be sampled from this queue, where k is less than C. As an illustrative example, the similarity between the two can be calculated using the following formula:

[0063]

[0064] Among them, w i,j Given the first auxiliary feature sample index i, this represents the scene similarity between the sample with index j in the second target feature sample set. h(.) represents the feature distance function between the two feature samples, and C is the length of the queue.

[0065] It is understandable that for multiple first auxiliary feature samples y within the first auxiliary feature sample set... op,1 ~y op,N Other given auxiliary feature samples in the set can be similarly derived from multiple target feature samples y' within the second target feature sample set. in,1 ~y' in,N The process involves selecting corresponding target feature samples to form more positive sample pairs, and selecting samples based on sample similarity to form more negative sample pairs, which will not be elaborated upon here.

[0066] On the other hand, refer to Figure 4 The latter two image feature extraction channels shown describe the construction process of additional positive and negative sample pairs given a target feature sample. Specifically, for a given first auxiliary feature sample among multiple first auxiliary feature samples:

[0067] (1) The process of constructing additional positive sample pairs may include: selecting second auxiliary feature samples from multiple second auxiliary feature samples that correspond to a given first target feature sample to form additional positive sample pairs. For example, combining Figure 4 As shown, for multiple first target feature samples y in the first target feature sample set in,1 ~y in,N Given the first target feature sample y in,i It can be derived from multiple second auxiliary feature samples y' within the second auxiliary feature sample set. op,1 ~y' op,N Select the corresponding auxiliary feature sample y' op,i , thus making y in,i With y' op,i This constitutes a positive sample pair.

[0068] (2) The process of constructing additional negative sample pairs may include: selecting multiple second auxiliary feature samples from multiple candidate second auxiliary feature samples that do not correspond to a given first target feature sample to form multiple additional negative sample pairs. For example, combining Figure 4 As shown, for multiple first target feature samples y in the first target feature sample set in,1 ~y in,N Given target feature sample y in,i We can first select from multiple second auxiliary feature samples y' within the second auxiliary feature sample set. op,1 ~y' op,N Auxiliary feature samples y' that already form positive sample pairs are excluded. op,i As multiple candidate second auxiliary feature samples, auxiliary feature samples are selected from them to form multiple additional negative sample pairs.

[0069] It is understandable that, in order to overcome the problem of selecting samples with similar scenes to construct negative sample pairs, the construction of negative sample pairs can also be guided by the same idea of ​​weighted sampling based on sample similarity. For example, sampling can be performed based on the direct similarity between each candidate second auxiliary feature sample and the given first target feature sample; or sampling can be performed based on the similarity between each candidate second auxiliary feature sample and the corresponding first auxiliary feature sample of the given first target feature sample.

[0070] The following combination Figure 5B Specifically, this describes the similarity calculation of feature samples given a first target feature sample, and the process of constructing negative sample pairs based on weighted sampling. Combined with... Figure 5B Description: For a given first target feature sample y in the first target feature sample set... in,i It is possible to find the first auxiliary feature sample y corresponding to it in the first auxiliary feature sample set. opi Then, calculate a second candidate auxiliary sample y' within the second auxiliary feature sample set. op,j The corresponding auxiliary feature sample y opi i The similarity between the samples is used to determine the candidate auxiliary sample y'. op,j The selected weights prevent feature samples with similar scenes from being chosen to form negative sample pairs. For example, y' with relatively low similarity can be selected. op,2 y' op,j and y' op,N , thus making y in,i With y' op,2 y in,i With y' op,j y in,i With y' op,N Three additional negative sample pairs are formed. Optionally, in this disclosure, a queue of length C can be set in the second auxiliary feature sample set. When the queue is full, the old samples will be deleted. Accordingly, during the construction of the negative sample pairs, k negative samples can be sampled from this queue, where k is less than C.

[0071] It is understandable that for multiple first target feature samples y within the first target feature sample set... in,1 ~y in,N Other given first target feature samples can be similarly derived from multiple second auxiliary feature samples y' within the second auxiliary feature sample set. op,1 ~y' op,N The process involves selecting corresponding target feature samples to form more additional positive sample pairs, and selecting samples based on sample similarity guidance to form more additional negative sample pairs, which will not be elaborated upon here.

[0072] The image feature sample expansion method according to the embodiments of this disclosure expands the original single target image feature sample by introducing auxiliary image data, and guides the construction process of negative sample pairs based on sample feature similarity rules, thereby further overcoming the sample scene similarity problem in the sample pair construction process, so as to provide more suitable training sample data for learning.

[0073] Method for constructing a neural network

[0074] Figure 6 A flowchart illustrating a method for constructing a neural network for image processing according to an embodiment of the present disclosure is shown. Figure 7 A schematic diagram illustrating an example of constructing a neural network for image processing according to embodiments of the present disclosure is shown. The following details the specific construction of the neural network. Figure 6 and Figure 7 This disclosure describes a method for constructing a neural network for image processing according to embodiments of the present disclosure. In the embodiments of the present disclosure, the parameters of the neural network may be initially set or learned to a certain extent. To improve the performance of the neural network in downstream tasks, it can be continuously trained and learned to enhance its performance. For example, after the neural network is constructed, it can be applied to various practical application scenarios such as semantic segmentation and object detection based on target image data; this disclosure is not limited thereto. For example, the trained network can be used in scenarios such as defect detection of photovoltaic panels, road object detection, and medical image analysis.

[0075] It is understood that the construction and training of the neural network disclosed herein can be based on training sample pairs generated by the image feature training sample augmentation techniques discussed above. Therefore, the relevant content will be briefly described below, while detailed descriptions of the same details will be omitted.

[0076] like Figure 6 As shown, in step S601, a neural network including a target encoder and an auxiliary encoder is constructed. It is understood that the neural network of this disclosure may include other network layers besides the target encoder and auxiliary encoder, and the neural network may have different network structures and more or fewer network layers; this disclosure is not limited thereto.

[0077] In step S602, the target encoder and the auxiliary encoder are respectively configured to extract multiple target feature samples and multiple corresponding auxiliary feature samples from multiple target images and multiple corresponding auxiliary images. As an illustrative example, the target images can be infrared images, and the auxiliary images can be visible light images. For any one of the multiple infrared images, there exists a visible light image among the multiple visible light images that captures the same scene as the infrared image. Therefore, based on this correspondence, multiple corresponding auxiliary images can be determined for multiple target images.

[0078] Additionally, in step S602, an encoder can be used to extract features from the target image and the corresponding auxiliary image to obtain target feature samples and corresponding auxiliary feature samples. As an illustrative example, this encoder can be part of a deep convolutional neural network. Continuing with... Figure 7 The description can utilize the auxiliary feature encoder f α Extract auxiliary feature samples y from N auxiliary images respectively. op,1 ~y op,N This serves as an auxiliary feature sample set. Similarly, the target feature encoder f can be used. β Extract target feature samples y from N target images respectively. in,1 ~y in,N This serves as the target feature sample set. It should be noted that, although... Figure 7 The image shows two auxiliary feature sample sets, but the image feature samples in these two sets are identical, both generated by the auxiliary feature encoder f. α The two target feature sample sets are generated in the same way. Based on this, a sample can be drawn from the auxiliary feature sample set and the target feature sample set respectively to construct positive and negative sample pairs for multi-view contrastive learning.

[0079] return Figure 6 In step S603, for a given auxiliary feature sample among the plurality of auxiliary feature samples, feature samples are selected from the plurality of target feature samples to form a positive sample pair and multiple negative sample pairs. For example, the construction of positive and negative sample pairs given an auxiliary feature sample mainly includes the following two aspects:

[0080] (1) Select the target feature sample corresponding to the given auxiliary feature sample from the plurality of target feature samples to form the positive sample pair. For example... Figure 7 As shown, for multiple auxiliary feature samples y in the auxiliary feature sample set op,1 ~y op,N Given auxiliary feature sample y op,i It can be derived from multiple target feature samples y within the target feature sample set. in,1 ~yin,N Select the corresponding target feature sample y in,i , thus making y op,i With y in,i This constitutes a positive sample pair.

[0081] (2) From the plurality of candidate target feature samples that do not correspond to the given auxiliary feature sample, multiple target feature samples are selected to form the plurality of negative sample pairs based on the similarity between each candidate target feature sample and the given auxiliary feature sample. Combined with... Figure 7 Description: For multiple auxiliary feature samples y in the auxiliary feature sample set... op,1 ~y op,N Given auxiliary feature sample y op,i We can first select from multiple target feature samples y within the target feature sample set. in,1 ~y in,N Exclude target feature samples y that already form positive sample pairs in,i Multiple candidate target feature samples are used to select target feature samples to form multiple negative sample pairs. In this disclosure, the scene similarity between two image feature samples to be selected can be evaluated based on the similarity between each candidate target feature sample and a given auxiliary feature sample, thereby sampling based on the calculated similarity, thereby avoiding the construction of two samples with similar scenes as negative sample pairs.

[0082] One approach is to directly utilize the similarity between each candidate target feature sample and a given auxiliary feature sample; that is, to directly calculate the candidate target feature sample y. in,1 ~y in,N (where y is excluded) in,i ) and given auxiliary feature sample y op,i The similarity between them is used to evaluate the feature samples y of each candidate target. in,1 ~y in,N (where y is excluded) in,i The corresponding scenarios and the given auxiliary feature samples y op,i The similarity between the corresponding scenes is used to avoid selecting samples with similar scenes to form negative sample pairs.

[0083] As another implementation, corresponding auxiliary feature samples for each candidate target feature sample can be determined, and multiple target feature samples can be selected to form the multiple negative sample pairs based on the similarity between each corresponding auxiliary feature sample and the given auxiliary feature sample. Figure 7 Description, for a certain candidate target feature sample y in,j It is possible to find its corresponding auxiliary feature sample y. op,j Then calculate the corresponding auxiliary feature sample y.op,j With a given auxiliary feature y op,i The similarity between the samples is used to determine the candidate target sample y. in,j The selected weights. For example, k target feature samples can be selected in ascending order of similarity to form k negative sample pairs.

[0084] It is understandable that for multiple auxiliary feature samples y op,1 ~y op,N Other given auxiliary feature samples in the dataset can be similarly derived from multiple target feature samples y. in,1 ~y in,N The process involves selecting corresponding target feature samples to form more positive sample pairs, and guiding the construction of negative sample pairs based on the similarity between samples. This will not be elaborated upon here.

[0085] return Figure 6 In step S604, for a given target feature sample among the plurality of target feature samples, feature samples are selected from the plurality of auxiliary feature samples to form an additional positive sample pair and a plurality of additional negative sample pairs. For example, the construction of positive and negative sample pairs for a given target feature sample mainly includes the following two aspects:

[0086] (1) Select an auxiliary feature sample from the plurality of auxiliary feature samples that corresponds to the given target feature sample to form an additional positive sample pair. For example, combining Figure 7 Description: For multiple target feature samples y in the target feature sample set... in,1 ~y in,N Given auxiliary feature sample y in,i It can be derived from multiple auxiliary feature samples y within the auxiliary feature sample set. op,1 ~y op,N Select the corresponding auxiliary feature sample y op,i , thus making y in,i With y op,i This constitutes a positive sample pair.

[0087] (2) From the plurality of auxiliary feature samples that do not correspond to the given target feature sample, multiple auxiliary feature samples are selected to form multiple additional negative sample pairs based on the similarity between each candidate auxiliary feature sample and the given target feature sample. Continue to combine... Figure 7 Description, for multiple target feature samples y in,1 ~y in,N Given target feature sample y in,i We can first start with multiple auxiliary feature samples y from the auxiliary feature sample set. op,1 ~y op,NAuxiliary feature samples y that already form positive sample pairs are excluded. op,i Multiple candidate auxiliary feature samples are used to select auxiliary feature samples to form multiple additional negative sample pairs. In this disclosure, the scene similarity between two image feature samples to be selected can be evaluated based on the similarity between each candidate auxiliary feature sample and a given target feature sample, thereby sampling based on the calculated similarity, thereby avoiding the construction of two samples with similar scenes as negative sample pairs.

[0088] One approach is to directly utilize the similarity between each candidate auxiliary feature sample and the given target feature sample; that is, to directly calculate the similarity between the candidate auxiliary feature samples y. op,1 ~y op,N (excluding y) op,i ) and given target feature sample y in,i The feature distance between them is used to evaluate each candidate auxiliary feature sample y. op,1 ~y op,N (excluding y) op,i The corresponding scenarios and the given target feature samples y in,i The similarity between the corresponding scenes is used to avoid selecting samples with similar scenes to form negative sample pairs.

[0089] One implementation approach is to determine the corresponding auxiliary feature samples of the given target feature sample, and select multiple auxiliary feature samples to constitute the multiple additional negative samples based on the similarity between each candidate auxiliary feature sample and its corresponding auxiliary feature sample. Combined with... Figure 7 Description, for a given target feature sample y in,i It is possible to find its corresponding auxiliary feature sample y. opi Then calculate a candidate auxiliary sample y op,j The corresponding auxiliary feature sample y opi i The similarity between the samples is used to determine the candidate auxiliary sample y. op,j The selected weights. For example, k target feature samples can be selected in ascending order of similarity to form k additional negative sample pairs.

[0090] return Figure 6In step S605, the loss of the neural network is calculated based on the positive sample pairs and the plurality of negative sample pairs, the additional positive sample pairs and the plurality of additional negative sample pairs. As an illustrative example, the neural network loss with reference to auxiliary features can be calculated based on the positive sample pairs and the plurality of negative sample pairs; and the neural network loss with reference to target features can be calculated based on the additional positive sample pairs and the plurality of additional negative sample pairs; then, the neural network loss with reference to auxiliary features and the neural network loss with reference to target features can be summed to obtain the overall loss of the neural network. The overall loss of the neural network can be calculated in various ways; for the sake of completeness, the NCE loss can be calculated using the following formula:

[0091]

[0092] in, Let represent the expected value, log represent the logarithmic function, h(.) represent the similarity function between two features, and k be the number of negative sample pairs. Furthermore, the first term in the above formula represents the neural network loss with reference to auxiliary features, while the second term represents the neural network loss with reference to the target feature.

[0093] In step S606, the parameters of the target encoder and the auxiliary encoder are updated based on the loss. As discussed above, the goal of multi-view learning is to map semantically adjacent variables more closely in the embedding space while pushing away irrelevant variables. To this end, the feature similarity of positive and negative sample pairs formed by the features extracted by the current target encoder and auxiliary encoder can be measured based on the calculated loss, and the network parameters can be updated accordingly, so that the overall loss of the neural network continuously decreases. In the embodiments of this disclosure, the training of the neural network can be carried out by inputting a certain number of auxiliary images and target images to update and iterate the parameters of the neural network, so as to make the distance between two samples in a positive sample pair smaller and the distance between two samples in a negative sample pair larger in the trained model, thereby performing self-supervised learning on unlabeled data. For example, the N auxiliary images and N target images mentioned above can be the amount of training data in a minibatch during the training process. According to embodiments of this disclosure, a loss term related to the feature similarity of sample pairs can be used as an important indicator to measure the performance of a neural network, thereby updating the parameters of the neural network based on the calculated loss, so that the neural network can perform better in extracting target image features.

[0094] For example, in the process of constructing the aforementioned neural network, the network parameters can be updated and the final neural network model to be used can be determined by minimizing the total loss of the neural network. Alternatively, in the embodiments of this disclosure, the construction of the neural network can be completed after a predetermined number of iterations to determine the final neural network model to be used.

[0095] On the other hand, similar combinations to the above Figure 4 As described, in this embodiment of the disclosure, the corresponding encoder can be configured based on the Mean Teacher concept to achieve better learning results. The following, in conjunction with... Figure 8 This describes an example of configuring a corresponding encoder using the Mean Teacher concept and learning based on it. For example, the target encoder described above includes a first target encoder and a second target encoder, wherein the first and second target encoders are respectively configured to extract multiple first target feature samples and multiple second target feature samples from the plurality of target images, and wherein the second target encoder is a weighted average version of the first target encoder. Similarly, the auxiliary encoder described above includes a first auxiliary encoder and a second auxiliary encoder, wherein the first and second auxiliary encoders are respectively configured to extract multiple first auxiliary feature samples and multiple second auxiliary feature samples from the plurality of auxiliary images, and wherein the second auxiliary encoder is a weighted average version of the first auxiliary encoder. Accordingly, the corresponding sample pairs can be constructed in the following manner.

[0096] On the one hand, for a given first auxiliary feature sample among multiple first auxiliary feature samples:

[0097] (1) The process of constructing a positive sample pair may include: selecting a second target feature sample corresponding to a given first auxiliary feature sample from multiple second target feature samples to form the positive sample pair.

[0098] (2) The process of constructing negative sample pairs may include: selecting multiple second target feature samples from multiple candidate second target feature samples that do not correspond to a given first auxiliary feature sample, based on the similarity between the corresponding second auxiliary feature sample of each candidate second target feature sample and the given first auxiliary feature sample, to form the multiple negative sample pairs.

[0099] On the other hand, for a given first target feature sample among multiple first target feature samples:

[0100] (1) The process of constructing an additional positive sample pair may include: selecting a second auxiliary feature sample corresponding to a given first target feature sample from multiple second auxiliary feature samples to form an additional positive sample pair.

[0101] (2) The process of constructing additional negative sample pairs may include: selecting multiple second auxiliary feature samples from multiple candidate second auxiliary feature samples that do not correspond to the given first target feature sample, based on the similarity between each candidate second auxiliary feature sample and the corresponding first auxiliary feature sample of the given first target feature sample, to form multiple additional negative sample pairs.

[0102] Accordingly, the overall loss of the neural network can be calculated based on the obtained positive and negative sample pairs. For the sake of completeness, the NCE loss can be calculated using the following formula:

[0103]

[0104] in, Let f represent the expected value, log represent the logarithmic function, h(.) represent the similarity function between two features, and k be the number of negative sample pairs. Furthermore, the first term in the above formula represents the neural network loss with reference to auxiliary features, while the second term represents the neural network loss with reference to the target features. After calculating the above losses, the parameters of the first target encoder and the first auxiliary encoder can be updated based on these losses. It is understood that it is not necessary to update the parameters of the second target encoder and the second auxiliary encoder based on the calculated losses; these two encoders can be updated according to the parameter update method in the MeanTeacher network. For example, the second auxiliary encoder f... α The parameter of ' can be the second auxiliary encoder f α The parameters of ' and the corresponding first auxiliary encoder f α The parameters are weighted, and therefore can be expressed according to the formula f discussed above. α '=εf α '+(1-ε)f α The relationship is used to update the second auxiliary encoder f. α The parameters of '. Similarly, the second target encoder f β The parameter of ' can be the second target encoder f β The parameters of ' and the corresponding first target encoder f β The parameters are weighted, and therefore can be expressed according to the formula f discussed above. β '=εf β '+(1-ε)f β The relationship is used to update the second target encoder f. β The parameters are as described above. It is understood that the update method described above is merely an example, and this disclosure does not limit the specific parameter update methods in the Mean Teacher network.

[0105] In addition, to achieve better training results, this disclosure not only uses positive and negative image feature sample pairs generated from unlabeled training data to learn the encoder parameters in multi-view contrastive learning, but also uses labeled data to train some auxiliary tasks to constrain the encoder, thereby preserving interest information for downstream tasks. Specifically, the following combines... Figure 9 The diagram illustrates the neural network learning concept that combines unsupervised and supervised learning proposed in this disclosure, which mainly includes the following steps:

[0106] (1) The first target encoder and the second target encoder, as well as the first auxiliary encoder and the second auxiliary encoder, are arranged in the unsupervised learning module of the neural network. It can be understood that the above combination... Figures 6-8 The neural network discussed is constructed and trained using a multi-view comparative learning process under unlabeled data conditions, and therefore can be placed within the unsupervised learning module of the entire neural network. Accordingly, to obtain better training, labeled training data can be used to constrain the phased learning results (i.e., encoder parameters) obtained during the unsupervised learning process.

[0107] (2) A supervised learning module is constructed in the neural network, and the updated parameters of the first target encoder and the first auxiliary encoder are passed from the unsupervised learning module to the supervised learning module. For example, it can be done as follows: Figures 6-8 The discussion focuses on multi-view contrastive learning under unlabeled data conditions, after one or more iterations (e.g., Figure 9 As shown in the left-hand box diagram), the encoder parameters obtained after phased training are passed to the supervised learning module (such as...). Figure 9 (As shown by the right arrow between the two box diagrams on the left and right sides) so that it can be constrained based on the auxiliary task.

[0108] (3) In the supervised learning module, the updated parameters of the first target encoder and the first auxiliary encoder are used to extract target image features and / or auxiliary image features from the target image and / or auxiliary image with known labels to perform auxiliary tasks.

[0109] It is understood that auxiliary tasks can include either or both of auxiliary tasks based on auxiliary images and auxiliary tasks based on target images. It should be noted that both the auxiliary and target images used in the auxiliary task carry known artificial labels, such as data from the ImageNet dataset. This allows the loss of the auxiliary task to be calculated based on the prediction results and the known labels, thereby updating the relevant parameters. For example, as... Figure 9 As shown in the box on the right, the auxiliary task based on the auxiliary image can be performed using an auxiliary encoder f.α The classifier, which serves as the backbone, is used to predict the classification results of auxiliary images, thereby calculating the auxiliary task loss based on the known labels of the auxiliary images and the prediction results. Similarly, auxiliary tasks based on the target image can be performed using a target encoder f. β The classifier, which forms the backbone, is used to predict the target image and provide corresponding classification results. Based on the known labels and prediction results, it calculates the auxiliary task loss based on the target data. The following combines... Figure 10 This describes the specific details related to auxiliary tasks in the supervised learning module.

[0110] As mentioned above, the auxiliary task based on the auxiliary image can be performed using an auxiliary encoder f. α A classifier with a main body, for example, which, in addition to including an auxiliary encoder f α In addition, it includes an additional network layer f based on auxiliary data for auxiliary perception. α,aux Correspondingly, the auxiliary encoder f, after receiving phased updates from the unsupervised learning module... α After obtaining the parameters, the auxiliary encoder f can be... α Used to extract auxiliary image features y op And by using additional network layers f α,aux Features are further extracted from these features and used in auxiliary tasks based on auxiliary images. Similarly, auxiliary tasks based on target images can be performed using a target encoder f. β A classifier with a main body, for example, which, in addition to including the target encoder f β In addition, it includes an additional network layer f based on the target data and the target's perception. β,aux Correspondingly, the target encoder f, after receiving phased updates from the unsupervised learning module... β After obtaining the parameters, the target encoder f can be... β Used to extract target image features y in And by using additional network layers f β,aux Further features are extracted from them and then used in auxiliary tasks based on target images.

[0111] It should be noted that the labeled image data used in the auxiliary task of the supervised learning module can come from the same or different domains as the unlabeled image data used in the unsupervised learning module (therefore, the training image data used in the two modules are unrelated). For example, these image data can come from photovoltaic panel scenes, road scenes, medical image scenes, etc. This disclosure does not restrict the domain of the labeled image data in the supervised learning module. Furthermore, the labeled target image data and auxiliary image data used in the auxiliary task of the supervised learning module can also be unrelated to each other; for example, the two types of image data do not need to satisfy the one-to-one correspondence discussed above. Moreover, although the above example uses a classification task as an auxiliary task, the auxiliary task is not limited to this. For example, the auxiliary task can also include other tasks such as segmentation and object detection based on the corresponding type of image data, as long as the auxiliary task can constrain the learning of the encoder parameters.

[0112] (4) In the supervised learning module, the loss of the auxiliary task is calculated based on the execution result of the auxiliary task and the known labels. For example, as described above in conjunction with point (3), the auxiliary task based on the auxiliary image can be performed using an auxiliary encoder f. α The classifier with the main body is used to calculate the auxiliary task loss based on the known labels of the auxiliary images and the prediction results. Similarly, the auxiliary task based on the target image can be performed using a target encoder f. β The classifier serves as the backbone, thereby calculating the auxiliary task loss based on the known labels and prediction results. Various known methods can be used to calculate the auxiliary task loss, and this disclosure does not limit this approach.

[0113] (5) In the supervised learning module, the parameters of the first target encoder and / or the first auxiliary encoder are updated according to the loss of the auxiliary task, and the updated parameters of the first target encoder and / or the first auxiliary encoder are passed back to the unsupervised learning module. For example, as described above in conjunction with points (3) and (4), the auxiliary task loss based on the auxiliary data and the auxiliary task loss based on the target data can be obtained respectively. Based on this, since the auxiliary image data and the target image data do not need to be related to each other, the corresponding network parameters can be updated separately based on these two losses. For example, the auxiliary encoder f can be updated using the auxiliary task loss based on the auxiliary data. α and its additional network layer f α,aux The parameters; or the target encoder f can be updated using the auxiliary task loss based on the target data. β and its additional network layer f β,auxThe parameters. Alternatively, these two losses can be combined to update the network layer parameters in a combined manner. This disclosure does not limit the specific parameter update process in the auxiliary task. Subsequently, after one or several iterations in the auxiliary task in the supervised learning module, the updated parameters of the encoder obtained after phased training (i.e., the auxiliary encoder f) can be updated. α and target encoder f β The updated parameters are returned to the unsupervised learning module, and then the next iteration of multi-view contrastive learning is performed in the unsupervised learning module. This process can be repeated until the updated target encoder f is reached. β Meets expected performance

[0114] It should be noted that, due to the additional network layer parameters f in the supervised learning module after training... α,aux and f β,aux This will be abandoned, therefore the auxiliary task does not need to converge; it only needs to be able to update the auxiliary encoder f during the auxiliary task. α and target encoder f β The parameters can be constrained.

[0115] The method for constructing a neural network according to embodiments of this disclosure expands the original single target image feature samples by introducing auxiliary image data, thereby enabling network construction and learning based on multi-view image data. Furthermore, by guiding the construction process of negative sample pairs based on sample feature similarity rules, the problem of difficulty in learning a good neural network due to sample scene similarity is further overcome. This allows the neural network to provide good performance even in scenarios where the differences between image data samples may not be significant after multi-view comparative learning.

[0116] Device for augmenting image feature samples

[0117] According to another aspect of this disclosure, an apparatus for expanding image feature samples is provided, which is described below in conjunction with... Figure 11 Please describe the device 1100 in detail.

[0118] Figure 11 A hardware block diagram of a device according to an embodiment of the present disclosure is shown. Figure 11 As shown, device 1100 includes processor U1101 and memory U1102.

[0119] The processor U1101 can be any processing-capable device capable of implementing the functions of the various embodiments of this disclosure. For example, it can be a general-purpose processor, digital signal processor (DSP), ASIC, field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.

[0120] The memory U1102 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may also include other removable / non-removable, volatile / non-volatile computer system memory, such as hard disk drives, floppy disks, CD-ROMs, DVD-ROMs, or other optical storage media.

[0121] In this embodiment, the memory U1102 stores computer program instructions, and the processor U1101 can execute the instructions stored in the memory U1102. When the computer program instructions are executed by the processor, the processor performs the method for expanding image feature samples according to this embodiment. The method for expanding image feature samples is similar to that described above. Figures 1-5B The descriptions are essentially the same, so to avoid repetition, they will not be repeated. Examples of devices could include computers, servers, workstations, and so on.

[0122] According to another aspect of this disclosure, an apparatus for expanding image feature samples is provided, which is described below in conjunction with... Figure 12 Please describe the device 1200 in detail.

[0123] Figure 12 A structural block diagram of an apparatus for expanding image feature samples according to an embodiment of the present disclosure is shown. Figure 12 As shown, the device 1200 includes a feature sample extraction unit U1201 and a sample pair construction unit U1202. Each component can respectively perform the functions described above. Figures 1-5B The various steps / functions of the method for expanding image feature samples are described below. Therefore, to avoid repetition, only a brief description of the device is given below, while detailed descriptions of the same details are omitted.

[0124] The feature sample extraction unit U1201 can determine multiple auxiliary images corresponding to multiple target images, and extract multiple target feature samples and multiple auxiliary feature samples from the multiple target images and the corresponding multiple auxiliary images.

[0125] The sample pair construction unit U1202 can, for a given auxiliary feature sample among the plurality of auxiliary feature samples, select feature samples from the plurality of target feature samples to form a positive sample pair and multiple negative sample pairs. Specifically, the sample pair construction unit U1202 may include a positive sample pair construction subunit and a negative sample pair construction subunit, wherein: the positive sample pair construction subunit can select target feature samples corresponding to the given auxiliary feature sample from the plurality of target feature samples to form the positive sample pair; the negative sample pair construction subunit can select multiple target feature samples from multiple candidate target feature samples that do not correspond to the given auxiliary feature sample, based on the similarity between the corresponding auxiliary feature samples of each candidate target feature sample and the given auxiliary feature sample, to form the multiple negative sample pairs. It can be understood that the positive sample pairs and the multiple negative sample pairs are used to train a neural network for extracting image features, for example, for multi-view contrast learning.

[0126] According to the device for expanding image feature samples disclosed herein, the original single target image feature sample is expanded by introducing auxiliary image data, and the construction process of negative sample pairs is guided based on sample feature similarity rules, thereby further overcoming the sample scene similarity problem in the sample pair construction process, so as to provide training sample data that is more suitable for learning.

[0127] Device for constructing a neural network

[0128] According to another aspect of this disclosure, an apparatus for constructing a neural network is provided, which is described below in conjunction with Figure 13 The device 1300 for constructing this neural network is described in detail.

[0129] Figure 13 A hardware block diagram of a construction device according to an embodiment of the present disclosure is shown. Figure 13 As shown, the construction device 1300 includes a processor U1301 and a memory U1302.

[0130] The processor U1301 can be any processing-capable device capable of implementing the functions of the various embodiments of this disclosure. For example, it can be a general-purpose processor, digital signal processor (DSP), ASIC, field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.

[0131] The memory U1302 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may also include other removable / non-removable, volatile / non-volatile computer system memory, such as hard disk drives, floppy disks, CD-ROMs, DVD-ROMs, or other optical storage media.

[0132] In this embodiment, the memory U1302 stores computer program instructions, and the processor U1301 can execute the instructions stored in the memory U1302. When the computer program instructions are executed by the processor, the processor performs a method for constructing a neural network according to an embodiment of this disclosure. This method for constructing a neural network is related to the method described above. Figures 6-10 The descriptions are essentially the same, so to avoid repetition, they will not be repeated. Examples of such construction devices could include computers, servers, workstations, and so on.

[0133] According to another aspect of this disclosure, an apparatus for constructing a neural network is provided, in conjunction with the following... Figure 14 The construction device 1400 is described in detail.

[0134] Figure 14 A structural block diagram of a device for constructing a neural network according to an embodiment of the present disclosure is shown. Figure 14 As shown, the construction device 1400 includes a feature sample extraction unit U1401, a sample pair construction unit U1402, a loss calculation unit U1403, and a neural network update unit U1404. Each component can respectively perform the functions described above. Figures 6-10 The methods for constructing neural networks are described below, and therefore, to avoid repetition, only a brief description of the device is given below, omitting detailed descriptions of the same details. It is understood that the constructed neural network may include a target encoder and an auxiliary encoder, etc.

[0135] The feature sample extraction unit U1401 can configure the target encoder and the auxiliary encoder to extract multiple target feature samples and multiple corresponding auxiliary feature samples from multiple target images and multiple corresponding auxiliary images, respectively.

[0136] The sample pair construction unit U1402 can, for a given auxiliary feature sample among the plurality of auxiliary feature samples, select feature samples from the plurality of target feature samples to form a positive sample pair and multiple negative sample pairs. Specifically, the sample pair construction unit U1402 may include a positive sample pair construction subunit and a negative sample pair construction subunit, wherein: the positive sample pair construction subunit can select target feature samples corresponding to the given auxiliary feature sample from the plurality of target feature samples to form the positive sample pair; the negative sample pair construction subunit can select multiple target feature samples from multiple candidate target feature samples that do not correspond to the given auxiliary feature sample from the plurality of target feature samples to form the multiple negative sample pairs based on the similarity between each candidate target feature sample and the given auxiliary feature sample. In addition, the sample pair construction unit U1402 can also, for a given target feature sample among the plurality of target feature samples, select feature samples from the plurality of auxiliary feature samples to form an additional positive sample pair and multiple additional negative sample pairs. Correspondingly, the positive sample pair construction subunit can select auxiliary feature samples corresponding to the given target feature sample from the plurality of auxiliary feature samples to form an additional positive sample pair; and the negative sample pair construction subunit can select multiple auxiliary feature samples from multiple candidate auxiliary feature samples that do not correspond to the given target feature sample from the plurality of auxiliary feature samples to form multiple additional negative sample pairs based on the similarity between each candidate auxiliary feature sample and the given target feature sample.

[0137] The loss calculation unit U1403 can calculate the loss of the neural network based on the positive sample pairs and the plurality of negative sample pairs, the additional positive sample pairs and the plurality of additional negative sample pairs.

[0138] The neural network update unit U1404 can update the parameters of the target encoder and the auxiliary encoder based on the loss.

[0139] According to the apparatus for constructing neural networks disclosed herein, auxiliary image data is introduced to expand the original single target image feature samples, thereby enabling network construction and learning based on multi-view image data. Furthermore, by guiding the construction process of negative sample pairs based on sample feature similarity rules, the problem of difficulty in learning a good neural network due to sample scene similarity is further overcome. This allows the neural network to provide good performance even in scenarios where the differences between image data samples may not be significant after multi-view comparative learning.

[0140] Computer readable storage medium

[0141] The fine-grained classification technique based on the generative adversarial network concept disclosed herein can also be implemented by providing a computer program product containing program code implementing the method or device, or by any storage medium storing such a computer program product.

[0142] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0143] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0144] Additionally, as used herein, the “or” used in a list of items beginning with “at least one” indicates a separate list, such that a list of, for example, “at least one of A, B, or C” means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word “exemplary” does not imply that the described example is preferred or better than other examples.

[0145] It should also be noted that in the apparatus and method of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0146] It will be understood by those skilled in the art that all or any part of the methods and apparatus of this disclosure can be implemented in hardware, firmware, software, or a combination thereof in any computing device (including processors, storage media, etc.) or network of computing devices. The hardware may be a general-purpose processor, digital signal processor (DSP), ASIC, field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, it may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration. The software may reside in any form of computer-readable tangible storage medium. By way of example and not limitation, such computer-readable tangible storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible by a computer. If used herein, the disks include compact discs (CDs), laser discs, optical discs, digital universal discs (DVDs), floppy disks, and Blu-ray discs.

[0147] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0148] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0149] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for expanding image feature samples, comprising: For multiple target images, determine multiple corresponding auxiliary images, and extract multiple target feature samples and multiple corresponding auxiliary feature samples from the multiple target images and the multiple corresponding auxiliary images; For a given auxiliary feature sample among the plurality of auxiliary feature samples, feature samples are selected from the plurality of target feature samples to form a positive sample pair and multiple negative sample pairs, wherein: The positive sample pair is formed by selecting target feature samples corresponding to the given auxiliary feature sample from the plurality of target feature samples. From the plurality of target feature samples that do not correspond to the given auxiliary feature sample, multiple target feature samples are selected to form the plurality of negative sample pairs based on the similarity between the corresponding auxiliary feature sample of each candidate target feature sample and the given auxiliary feature sample. The positive sample pairs and the plurality of negative sample pairs are used to train a neural network for extracting image features, wherein: Extracting multiple target feature samples from the multiple target images includes: Multiple first target feature samples are extracted using a first target encoder, and multiple second target feature samples are extracted using a second target encoder, wherein the second target encoder is a weighted average version of the first target encoder. Extracting multiple auxiliary feature samples from the multiple auxiliary images includes: Multiple first auxiliary feature samples are extracted using a first auxiliary encoder, and multiple second auxiliary feature samples are extracted using a second auxiliary encoder, wherein the second auxiliary encoder is a weighted version of the first auxiliary encoder. For a given auxiliary feature sample among the plurality of auxiliary feature samples, selecting feature samples from the plurality of target feature samples to form a positive sample pair and multiple negative sample pairs includes: For a given first auxiliary feature sample among the plurality of first auxiliary feature samples: The positive sample pair is formed by selecting a second target feature sample corresponding to the given first auxiliary feature sample from the plurality of second target feature samples. From the plurality of candidate second target feature samples that do not correspond to the given first auxiliary feature sample, a plurality of second target feature samples are selected to form the plurality of negative sample pairs based on the similarity between the corresponding second auxiliary feature sample of each candidate second target feature sample and the given first auxiliary feature sample.

2. The method according to claim 1, further comprising: For a given target feature sample among the plurality of target feature samples: From the plurality of auxiliary feature samples, select the auxiliary feature sample corresponding to the given target feature sample to form an additional positive sample pair, and From the plurality of auxiliary feature samples that do not correspond to the given target feature sample, multiple auxiliary feature samples are selected to form multiple additional negative sample pairs based on the similarity between each candidate auxiliary feature sample and the corresponding auxiliary feature sample of the given target feature sample. The additional positive sample pairs and the plurality of additional negative sample pairs are also used to train the neural network for extracting image features.

3. The method according to claim 1, further comprising: For a given first target feature sample among the plurality of first target feature samples: From the plurality of second auxiliary feature samples, a second auxiliary feature sample corresponding to the given first target feature sample is selected to form an additional positive sample pair, and From the plurality of candidate second auxiliary feature samples that do not correspond to the given first target feature sample, multiple second auxiliary feature samples are selected to form multiple additional negative sample pairs based on the similarity between each candidate second auxiliary feature sample and the corresponding first auxiliary feature sample of the given first target feature sample. The additional positive sample pairs and the plurality of additional negative sample pairs are also used to train the neural network for extracting image features.

4. The method according to any one of claims 1-3, wherein, The target image includes an infrared image, and the auxiliary image includes a visible light image.

5. A method for constructing a neural network for image processing, comprising: Construct a neural network that includes a target encoder and an auxiliary encoder; The target encoder and the auxiliary encoder are respectively configured to extract multiple target feature samples and multiple corresponding auxiliary feature samples from multiple target images and multiple corresponding auxiliary images; For a given auxiliary feature sample among the plurality of auxiliary feature samples, feature samples are selected from the plurality of target feature samples to form a positive sample pair and multiple negative sample pairs, wherein: The positive sample pair is formed by selecting target feature samples corresponding to the given auxiliary feature sample from the plurality of target feature samples. From the plurality of target feature samples that do not correspond to the given auxiliary feature sample, multiple target feature samples are selected to form the plurality of negative sample pairs based on the similarity between each candidate target feature sample and the given auxiliary feature sample. For a given target feature sample among the plurality of target feature samples, feature samples are selected from the plurality of auxiliary feature samples to form an additional positive sample pair and a plurality of additional negative sample pairs, wherein: From the plurality of auxiliary feature samples, select the auxiliary feature sample corresponding to the given target feature sample to form an additional positive sample pair, and From the multiple auxiliary feature samples that do not correspond to the given target feature sample, multiple auxiliary feature samples are selected to form multiple additional negative sample pairs based on the similarity between each candidate auxiliary feature sample and the given target feature sample. The loss of the neural network is calculated based on the positive sample pairs and the plurality of negative sample pairs, the additional positive sample pairs and the plurality of additional negative sample pairs; and Based on the loss, the parameters of the target encoder and the auxiliary encoder are updated. in: The target encoder includes a first target encoder and a second target encoder, wherein the first target encoder and the second target encoder are respectively configured to extract multiple first target feature samples and multiple second target feature samples from the plurality of target images, and wherein the second target encoder is a model weighted average version of the first target encoder. The auxiliary encoder includes a first auxiliary encoder and a second auxiliary encoder, wherein the first auxiliary encoder and the second auxiliary encoder are respectively configured to extract multiple first auxiliary feature samples and multiple second auxiliary feature samples from the plurality of auxiliary images, and wherein the second auxiliary encoder is a weighted average version of the first auxiliary encoder. For a given auxiliary feature sample among the plurality of auxiliary feature samples, selecting feature samples from the plurality of target feature samples to form a positive sample pair and multiple negative sample pairs includes: For a given first auxiliary feature sample among the plurality of first auxiliary feature samples: The positive sample pair is formed by selecting a second target feature sample corresponding to the given first auxiliary feature sample from the plurality of second target feature samples. From the plurality of candidate second target feature samples that do not correspond to the given first auxiliary feature sample, multiple second target feature samples are selected to form the plurality of negative sample pairs based on the similarity between the corresponding second auxiliary feature sample of each candidate second target feature sample and the given first auxiliary feature sample. For a given target feature sample among the plurality of target feature samples, selecting feature samples from the plurality of auxiliary feature samples to form an additional positive sample pair and multiple additional negative sample pairs includes: For a given first target feature sample among the plurality of first target feature samples: An additional positive sample pair is formed by selecting a second auxiliary feature sample from the plurality of second auxiliary feature samples that corresponds to the given first target feature sample. From the plurality of candidate second auxiliary feature samples that do not correspond to the given first target feature sample, multiple second auxiliary feature samples are selected to form multiple additional negative sample pairs based on the similarity between each candidate second auxiliary feature sample and the corresponding first auxiliary feature sample of the given first target feature sample. And among them: Updating the parameters of the target encoder and the auxiliary encoder based on the loss includes updating the parameters of the first target encoder and the first auxiliary encoder based on the loss.

6. The method according to claim 5, wherein: Selecting multiple target feature samples to form the multiple negative sample pairs based on the similarity between each candidate target feature sample and the given auxiliary feature sample includes: Determine the corresponding auxiliary feature samples for each candidate target feature sample, and select multiple target feature samples to form the multiple negative sample pairs based on the similarity between each corresponding auxiliary feature sample and the given auxiliary feature sample. Based on the similarity between each candidate auxiliary feature sample and the given target feature sample, multiple auxiliary feature samples are selected to form multiple additional negative sample pairs, including: Determine the corresponding auxiliary feature samples for the given target feature sample, and select multiple auxiliary feature samples to form the multiple additional negative samples based on the similarity between each candidate auxiliary feature sample and the corresponding auxiliary feature sample.

7. The method according to claim 5, further comprising: The first target encoder and the second target encoder, as well as the first auxiliary encoder and the second auxiliary encoder, are arranged in the unsupervised learning module of the neural network; A supervised learning module is constructed in the neural network, and the updated parameters of the first target encoder and the first auxiliary encoder are passed from the unsupervised learning module to the supervised learning module; In the supervised learning module, the updated parameters of the first target encoder and the first auxiliary encoder are used to extract target image features and / or auxiliary image features from the target image and / or auxiliary image with known labels to perform auxiliary tasks. In the supervised learning module, the loss of the auxiliary task is calculated based on the execution result of the auxiliary task and the known labels; as well as In the supervised learning module, the parameters of the first target encoder and / or the first auxiliary encoder are updated according to the loss of the auxiliary task, and the updated parameters of the first target encoder and / or the first auxiliary encoder are passed back to the unsupervised learning module.

Images