A self-supervised network construction method for feature extraction of complex illumination images

Abstract

The present disclosure proposes a method for constructing a self-supervised network for complex illumination image feature extraction, comprising: obtaining a first image corresponding to a target object captured under fill light conditions, a second image corresponding to the target object captured under no fill light conditions, and a pre-trained initial image feature extraction network, adjusting the color matrix and gamma value of the second image according to the first image to obtain a third image, and then determining, according to the first image, annotated feature information corresponding to the third image, training the initial image feature extraction network according to the first image, the third image, and the annotated feature information corresponding to the third image to obtain a target image feature extraction network, and being able to jointly train the image feature extraction network with the first image captured under fill light conditions and the second image captured under no fill light conditions, thereby being able to effectively improve the image feature extraction capability of the image feature extraction network in complex illumination scenes.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a method for extracting complex illumination image features from a supervised network and constructing the same. Background Art

[0002] The lighting conditions in coal mine underground tunnel scenes are poor, and most of them are low-illuminance scenes. The lighting conditions in coal mine underground tunnel scenes are poor, and most of them are low-illuminance scenes. Therefore, it is difficult to extract features from images collected in coal mine underground tunnel scenes.

[0003] In the related technology, the generalization ability of the image feature extraction network in the feature extraction task in the coal mine underground tunnel scene is poor, and it cannot meet the image feature extraction needs in the coal mine underground tunnel scene. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the purpose of the present invention is to propose a method for constructing a self-supervised network for complex lighting image feature extraction, which can jointly train the image feature extraction network by using a first image captured under fill light conditions and a second image captured under no fill light conditions, thereby effectively improving the image feature extraction ability of the image feature extraction network in complex lighting scenes, thereby effectively improving the image feature extraction effect in complex lighting scenes.

[0006] To achieve the above-mentioned purpose, the present disclosure proposes a method for extracting complex illumination image features from a supervised network and constructing the method, including:

[0007] Acquire a first image corresponding to the target object captured under fill light conditions, a second image corresponding to the target object captured without fill light conditions, and a pre-trained initial image feature extraction network;

[0008] Adjusting the color matrix and gamma value of the second image according to the first image to obtain a third image;

[0009] Determining, based on the first image, annotation feature information corresponding to the third image;

[0010] An initial image feature extraction network is trained according to the first image, the third image and the annotated feature information corresponding to the third image to obtain a target image feature extraction network.

[0011] The complex illumination image feature extraction self-supervised network construction method provided by the present disclosure includes at least the following beneficial effects: by obtaining a first image corresponding to a target object captured under fill light conditions, a second image corresponding to the target object captured under no fill light conditions, and a pre-trained initial image feature extraction network, and then adjusting the color matrix and gamma value of the second image according to the first image to obtain a third image, and then determining the annotated feature information corresponding to the third image according to the first image, and training the initial image feature extraction network according to the first image, the third image, and the annotated feature information corresponding to the third image to obtain a target image feature extraction network, thereby, the image feature extraction network can be jointly trained with the first image captured under fill light conditions and the second image captured under no fill light conditions, thereby effectively improving the image feature extraction capability of the image feature extraction network in complex illumination scenes, thereby effectively improving the image feature extraction effect in complex illumination scenes.

[0012] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0014] Figure 1 It is a flowchart of a method for extracting complex illumination image features from a supervised network and constructing the method according to an embodiment of the present disclosure;

[0015] Figure 2 is a flowchart of a method for extracting complex illumination image features from a supervised network and constructing the same, proposed in another embodiment of the present disclosure;

[0016] Figure 3 It is a flowchart of a method for extracting complex illumination image features from a supervised network and constructing the same according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present disclosure, and are not to be construed as limitations of the present disclosure. On the contrary, the embodiments of the present disclosure include all changes, modifications, and equivalents that fall within the spirit and connotation of the appended claims.

[0018] Figure 1 It is a flowchart of a method for extracting complex illumination image features from a supervised network and constructing the method according to an embodiment of the present disclosure.

[0019] like Figure 1 As shown, the complex illumination image features are extracted from the supervised network construction method, including:

[0020] S101: Acquire a first image corresponding to a target object captured under fill light conditions, a second image corresponding to the target object captured without fill light conditions, and a pre-trained initial image feature extraction network.

[0021] The target object may be any device, area, etc. in the underground tunnel scene of a coal mine, without any restriction.

[0022] Among them, the fill light condition refers to the presence of a fill light source in the image acquisition scene during the image acquisition process (for example, mining lamps, car lights, tunnel lights, signal lights, indicator lights, camera fill lights, etc.). Correspondingly, the no fill light condition refers to the absence of a fill light source in the image acquisition scene during the image acquisition process.

[0023] That is to say, in the embodiment of the present disclosure, when there is a fill light source in the coal mine underground tunnel scene, a first image can be captured for the target object, and after adding a fill light source to the coal mine underground tunnel scene, a second image can be captured for the target object.

[0024] The pre-trained initial image feature extraction network refers to a pre-trained model with image feature extraction capabilities. The model can be a neural network model or a machine learning model, and there is no restriction on this.

[0025] S102: adjusting the color matrix and gamma value of the second image according to the first image to obtain a third image.

[0026] After acquiring a first image corresponding to the target object captured under fill light conditions and a second image corresponding to the target object captured without fill light conditions, the embodiment of the present disclosure can adjust the color matrix and gamma value of the second image with reference to the first image to obtain a third image.

[0027] The color matrix of an image is a matrix used to describe the color information of an image. The color matrix of an image consists of a red channel, a green channel, and a blue channel. The pixel value on each channel represents the color intensity on that channel. Adjusting the color matrix can change the color saturation, contrast, hue and other properties of the image, thereby achieving color enhancement, conversion or correction.

[0028] The gamma value of an image is a parameter used to describe the brightness information of an image. The brightness of the image can be adjusted by adjusting the gamma value of the image.

[0029] In the embodiment of the present disclosure, the first image is captured under fill light conditions. Since the color and brightness presentation effects of the first image captured under fill light conditions are better than those of the second image captured under no fill light conditions, the color matrix and gamma value of the second image can be adjusted with reference to the first image captured for the same target object.

[0030] In some embodiments, adjusting the color matrix of the second image based on the first image may be performed by determining the color difference between the first image and the second image, and adjusting the color matrix of the second image based on the color difference determined above by using a linear transformation, histogram matching or other method, so that the similarity between the adjusted color matrix and the color matrix of the first image is greater than a similarity threshold.

[0031] In some embodiments, adjusting the color matrix of the second image according to the first image can be performed by converting the first image and the second image into grayscale space respectively, determining the average brightness values ​​of the first image and the second image respectively, calculating the ratio of the change in gamma value according to the ratio between the average brightness values ​​of the first image and the second image, and adjusting the gamma value of the second image based on the ratio of the gamma value change.

[0032] S103: Determine, according to the first image, annotation feature information corresponding to the third image.

[0033] In the embodiment of the present disclosure, after the color matrix and gamma value of the second image are adjusted according to the first image to obtain the third image, the annotation feature information corresponding to the third image can be determined according to the first image.

[0034] Among them, the annotated feature information corresponding to the third image can be used to determine the convergence timing of the initial image feature extraction network during the subsequent training process of the initial image feature extraction network. The annotated feature information can be, for example, color features, texture features, feature descriptors, feature point maps, etc., without limitation.

[0035] In the disclosed embodiment, since the first image and the third image are both acquired with respect to the target object, the annotation feature information corresponding to the third image can be determined based on the first image. The image features corresponding to the first image can be extracted, and then the image features corresponding to the first image are used as the annotation feature information. Since the first image is acquired with respect to the target object under fill light conditions, when the image features of the first image are used as the annotation feature information, the correct target and direction can be provided for the subsequent initial image feature extraction network based on the image features of the first image, thereby effectively assisting in improving the model training effect of the subsequent initial image feature extraction network.

[0036] S104: training an initial image feature extraction network according to the first image, the third image and the annotated feature information corresponding to the third image to obtain a target image feature extraction network.

[0037] In the embodiment of the present disclosure, after determining the annotated feature information corresponding to the third image based on the first image, the initial image feature extraction network can be trained based on the first image, the third image and the annotated feature information corresponding to the third image until the initial image feature extraction network converges to obtain the target image feature extraction network.

[0038] In some embodiments, the initial image feature extraction network is trained based on the first image, the third image and the annotated feature information corresponding to the third image. The first image can be input into the initial image feature extraction network to obtain the first image feature output by the initial image feature extraction network, and then the third image is input into the initial image feature extraction network to obtain the second image feature output by the initial image feature extraction network. Then, based on a preset loss function, a first loss value between the first image feature and the annotated feature information is determined, and a second loss value between the second image feature and the annotated feature information is determined. Then, a weighted summation process is performed on the first loss value and the second loss value to determine a target loss value. Then, when the target loss value is less than or equal to a loss threshold, it is determined that the initial image feature extraction network has converged to obtain a target image feature extraction network.

[0039] In the disclosed embodiment, a first image corresponding to the target object captured under fill light conditions, a second image corresponding to the target object captured under no fill light conditions, and a pre-trained initial image feature extraction network are obtained, and then the color matrix and gamma value of the second image are adjusted according to the first image to obtain a third image, and then based on the first image, annotated feature information corresponding to the third image is determined, and the initial image feature extraction network is trained according to the first image, the third image, and the annotated feature information corresponding to the third image to obtain a target image feature extraction network. Thus, the image feature extraction network can be jointly trained with the first image captured under fill light conditions and the second image captured under no fill light conditions, thereby effectively improving the image feature extraction capability of the image feature extraction network in complex lighting scenes, thereby effectively improving the image feature extraction effect in complex lighting scenes.

[0040] Figure 2 It is a flowchart of a method for extracting complex illumination image features from a supervised network and constructing the same according to another embodiment of the present disclosure.

[0041] like Figure 2 As shown, the complex illumination image features are extracted from the supervised network construction method, including:

[0042] S201: Acquire a first image corresponding to a target object captured under fill light conditions, a second image corresponding to the target object captured without fill light conditions, and a pre-trained initial image feature extraction network.

[0043] S202: Adjust the color matrix and gamma value of the second image according to the first image to obtain a third image.

[0044] The specific description of S201 - S202 can be found in the above embodiment, which will not be repeated here.

[0045] S203: Performing N homography transformation processes on the first image to obtain N fourth images, wherein the value of N is a positive integer, and the processing scale of each homography transformation process is different.

[0046] That is to say, in the embodiment of the present disclosure, the first image may be subjected to homography transformation processing at various scales to obtain a fourth image obtained by each processing.

[0047] S204: Obtain a first initial feature map corresponding to each fourth image and a second initial feature map corresponding to the first image.

[0048] In the disclosed embodiment, after performing N homography transformations on the first image to obtain N fourth images, a first initial feature map corresponding to each fourth image and a second initial feature map corresponding to the first image can be obtained.

[0049] In the disclosed embodiment, obtaining a first initial feature map corresponding to the fourth image and a second initial feature map corresponding to the first image may be performed by respectively inputting the fourth image and the second image into a pre-trained initial image feature extraction network to obtain the first initial feature map corresponding to the fourth image and the second initial feature map corresponding to the first image output by the initial image feature extraction network, or obtaining the first initial feature map corresponding to the fourth image and the second initial feature map corresponding to the first image, or respectively processing the first image and the fourth image using a scale-invariant feature transformation algorithm to obtain the first initial feature map corresponding to the fourth image and the second initial feature map corresponding to the first image.

[0050] S205: Determine, according to the N first initial feature maps and the second initial feature maps, the annotation feature information corresponding to the third image.

[0051] After acquiring the first initial feature map corresponding to each fourth image and the second initial feature map corresponding to the first image, the embodiment of the present disclosure can determine the annotation feature information corresponding to the third image based on the N first initial feature maps and the second initial feature maps.

[0052] In some embodiments, based on N first initial feature maps and second initial feature maps, determining the annotation feature information corresponding to the third image can be done by fusing the first initial feature map and the second initial feature map to obtain a fused feature map, and using the fused feature map as the annotation feature information corresponding to the third image.

[0053] Optionally, in some embodiments, based on N first initial feature maps and second initial feature maps, the annotation feature information corresponding to the third image is determined, and each first initial feature map can be restored by performing inverse homography transformation to obtain a third initial feature map, and then the N third initial feature maps and the second initial feature maps are fused to obtain a fourth initial feature map, and then the confidence corresponding to each feature point in the fourth initial feature map is determined, and then when the confidence is greater than or equal to the confidence threshold, a feature map composed of feature points corresponding to at least one confidence is used as the annotation feature information.

[0054] That is to say, in the embodiment of the present disclosure, the first initial feature map may be subjected to an inverse homography transformation and restoration process of a corresponding scale to obtain a third feature map, and then N third initial feature maps and the second initial feature map are fused to obtain a fifth feature map, and then the confidence corresponding to each feature point in the fourth initial feature map is determined, and then the confidence is compared with a pre-set confidence threshold, and when the confidence is greater than or equal to the confidence threshold, a feature map consisting of feature points corresponding to at least one confidence is used as the annotation feature information.

[0055] S206: Perform affine transformation processing on the third image to obtain a fifth image.

[0056] After the embodiment of the present disclosure adjusts the color matrix and gamma value of the second image according to the first image to obtain the third image, the third image can be affine transformed to obtain the fifth image. Thus, based on the affine transformation of the third image, the image information can be effectively enriched, so that in the process of training the initial image feature extraction network based on the fifth image obtained by the transformation, the training effect of the initial image feature extraction network can be effectively assisted and improved.

[0057] S207: training an initial image feature extraction network according to the first image, the third image, the fifth image and the annotated feature information to obtain a target image feature extraction network.

[0058] After performing affine transformation processing on the third image to obtain the fifth image, the embodiment of the present disclosure can train an initial image feature extraction network based on the first image, the third image, the fifth image and the annotated feature information to obtain a target image feature extraction network.

[0059] In some embodiments, the initial image feature extraction network is trained according to the first image, the third image, the fifth image and the annotated feature information. The first image, the third image and the fifth image can be respectively input into the initial image feature extraction network to obtain the predicted feature information corresponding to the first image, the predicted feature information corresponding to the third image and the predicted feature information corresponding to the fifth image output by the initial image feature extraction network. Then, based on a preset loss function, a first loss value between the predicted feature information corresponding to the first image and the annotated feature information, a second loss value between the predicted feature information corresponding to the third image and the annotated feature information, and a third loss value between the predicted feature information corresponding to the fifth image and the annotated feature information can be respectively determined. Then, based on preset weights, the first loss value, the second loss value and the third loss value are weightedly summed to obtain a target loss value. When the target loss value is less than or equal to a preset loss threshold, it is determined that the initial image feature extraction network has converged to obtain a target image feature extraction network.

[0060] In the disclosed embodiment, a first image corresponding to a target object captured under fill light conditions, a second image corresponding to the target object captured under no fill light conditions, and a pre-trained initial image feature extraction network are obtained, and then the color matrix and gamma value of the second image are adjusted according to the first image to obtain a third image, and then the first image is subjected to N homography transformation processes to obtain N fourth images, and then a first initial feature map corresponding to each fourth image and a second initial feature map corresponding to the first image are obtained, and then the annotated feature information corresponding to the third image is determined according to the N first initial feature maps and the second initial feature maps, and then the third image is subjected to affine transformation process to obtain a fifth image, and then the initial image feature extraction network is trained according to the first image, the third image, the fifth image, and the annotated feature information to obtain a target image feature extraction network, thereby jointly training the image feature extraction network with the first image captured under fill light conditions and the second image captured under no fill light conditions, thereby effectively improving the image feature extraction capability of the image feature extraction network in complex lighting scenes, thereby effectively improving the image feature extraction effect in complex lighting scenes.

[0061] like Figure 3 As shown, the complex illumination image features are extracted from the supervised network construction method, including:

[0062] S301: Acquire a first image corresponding to a target object captured under fill light conditions, a second image corresponding to the target object captured without fill light conditions, and a pre-trained initial image feature extraction network.

[0063] S302: adjusting the color matrix and gamma value of the second image according to the first image to obtain a third image.

[0064] S303: Determine, according to the first image, annotation feature information corresponding to the third image.

[0065] S304: Perform affine transformation processing on the third image to obtain a fifth image.

[0066] The specific description of S301 - S304 can be found in the above embodiment, which will not be repeated here.

[0067] S305: Input the third image and the fifth image into the feature point extraction sub-model in the initial image feature extraction network respectively to obtain a first predicted feature map corresponding to the third image and a second predicted feature map corresponding to the fifth image output by the feature point extraction sub-model.

[0068] Among them, the feature point extraction sub-model is a feature point extraction branch in the initial image feature extraction network. The feature point extraction sub-model can be used to extract multiple feature points in the image (the multiple feature points can be expressed in the form of feature maps).

[0069] In the embodiment of the present disclosure, after adjusting the color matrix and gamma value of the second image according to the first image to obtain the third image, and performing affine transformation processing on the third image to obtain the fifth image, the third image and the fifth image can be respectively input into the feature point extraction sub-model in the initial image feature extraction network to obtain a first predicted feature map corresponding to the third image and a second predicted feature map corresponding to the fifth image output by the feature point extraction sub-model.

[0070] Optionally, in some embodiments, the third image and the fifth image are respectively input into a feature point extraction submodel in the initial image feature extraction network to obtain a first predicted feature map corresponding to the third image and a second predicted feature map corresponding to the fifth image output by the feature point extraction submodel. The first size information of the third image and the second size information of the fourth image can be determined, and the third image and the fifth image are respectively input into the first stacked convolution layer in the feature point extraction submodel to obtain a first feature map corresponding to the third image and a second feature map corresponding to the fifth image output by the first stacked convolution layer. Sub-pixel convolution processing is then performed on the first feature map and the second feature map respectively to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map. The size of the third feature map is adjusted according to the first size information to obtain the first predicted feature map, and the size of the fourth feature map is adjusted according to the second size information to obtain the second predicted feature map.

[0071] The first stacked convolutional layer may be composed of 256 3×3 convolutional layers and 65 1×1 convolutional layers stacked together.

[0072] That is to say, in the embodiment of the present disclosure, the third image and the fifth image can be respectively input into the first stacked convolution layer in the feature point extraction submodel to obtain the first feature map corresponding to the third image and the second feature map corresponding to the fifth image output by the first stacked convolution layer, and then the first feature map and the second feature map are respectively subjected to sub-pixel convolution processing to eliminate the featureless channels made by the first feature map and the second feature map to obtain the third feature map corresponding to the first feature map and the fourth feature map corresponding to the second feature map, and then the third feature map is reshaped according to the first size information to obtain the first predicted feature map, and then the fourth feature map is reshaped according to the second size information to obtain the second predicted feature map.

[0073] S306: Input the third image and the fifth image into the feature descriptor extraction submodel in the initial image feature extraction network respectively to obtain a first feature descriptor corresponding to the third image and a second feature descriptor corresponding to the fifth image output by the feature descriptor extraction submodel.

[0074] The feature descriptor extraction submodel is a feature descriptor extraction subbranch in the initial image feature extraction network, and the feature descriptor extraction submodel can be used to extract feature descriptors in an image.

[0075] In the embodiment of the present disclosure, after adjusting the color matrix and gamma value of the second image according to the first image to obtain the third image, and performing affine transformation processing on the third image to obtain the fifth image, the third image and the fifth image can be respectively input into the feature descriptor extraction submodel in the initial image feature extraction network to obtain the first feature descriptor corresponding to the third image and the second feature descriptor corresponding to the fifth image output by the feature descriptor extraction submodel.

[0076] Optionally, in some embodiments, the third image and the fifth image are respectively input into the feature descriptor extraction submodel in the initial image feature extraction network to obtain a first feature descriptor corresponding to the third image and a second feature descriptor corresponding to the fifth image output by the feature descriptor extraction submodel. The third image and the fifth image can be respectively input into the second stacked convolution layer in the feature descriptor extraction submodel to obtain a third feature descriptor corresponding to the third image and a fourth feature descriptor corresponding to the fourth image output by the second stacked convolution layer, and then the third feature descriptor and the fourth feature descriptor are respectively subjected to bicubic linear interpolation processing to obtain a fifth feature descriptor corresponding to the third feature descriptor and a sixth feature descriptor corresponding to the fourth feature descriptor, and then the fifth feature descriptor is regularized to obtain the first feature descriptor, and the sixth feature descriptor is regularized to obtain the second feature descriptor.

[0077] The second stacked convolutional layer may be composed of 256 3×3 convolutional layers and 256 1×1 convolutional layers stacked in sequence.

[0078] That is to say, in the embodiment of the present disclosure, the third image and the fifth image can be respectively input into the second stacked convolution layer in the feature descriptor extraction submodel to obtain the third feature descriptor corresponding to the third image and the fourth feature descriptor corresponding to the fourth image output by the second stacked convolution layer, and then the third feature descriptor and the fourth feature descriptor are respectively subjected to bicubic linear interpolation processing to obtain the fifth feature descriptor corresponding to the third feature descriptor and the sixth feature descriptor corresponding to the fourth feature descriptor, and then the fifth feature descriptor is regularized to obtain the first feature descriptor, and the sixth feature descriptor is regularized to obtain the second feature descriptor.

[0079] S307: training an initial image feature extraction network according to the first predicted feature map, the second predicted feature map, the labeled feature information, the first feature descriptor, the second feature descriptor, the first image, and the third image to obtain a target image feature extraction network.

[0080] Optionally, in some embodiments, an initial image feature extraction network is trained based on the first predicted feature map, the second predicted feature map, the labeled feature information, the first feature descriptor, the second feature descriptor, the first image and the third image to obtain a target image feature extraction network. This may be by determining first loss information between the first image and the third image, then determining second loss information between the first predicted feature map and the labeled feature information, then determining third loss information between the first feature descriptor and the second feature descriptor, then determining fourth loss information between the second predicted feature map and the labeled feature information, and training the initial image feature extraction network based on the first loss information, the second loss information, the third loss information and the fourth loss information to obtain the target image feature extraction network.

[0081] The first loss information is used to describe the difference in pixel intensity between the first image and the third image. The determination method of the first loss information can be expressed as:

[0082]

[0083] Among them, L e (E, E′) is the first loss information, E is the third image, E′ is the first image, e hw is the pixel intensity value of the third image at position (h, w), e′ hw is the pixel intensity value of the first image at position (h, w), H and W are the pixel height and width of the first image.

[0084] The second loss information is used to describe the difference in feature point distribution between the first predicted feature map and the labeled feature information. The second loss information can be determined as follows:

[0085]

[0086] Among them, L p (X,Y) is the second loss information, X is the first predicted feature map, Y is the labeled feature information, A and B are the height and width of the third image, (a,b) represents the position coordinates of a feature point. Among them, X aby represents the predicted value of the feature probability at the corresponding channel position at the label y, X abk represents the predicted value of the k-channel position in the feature point probability map, Represents the sample X in the region ab The probability of belonging to a feature.

[0087] The third loss information is used to describe the distribution difference between the first feature descriptor and the second feature descriptor. The determination method of the third loss information can be expressed as:

[0088]

[0089] Where D is the first feature descriptor, D′ is the second feature descriptor, S is the position similarity function between the first feature descriptor and the second feature descriptor, and d ab Denotes the feature descriptor unit of the first feature descriptor, d′ a′b′ represents the feature descriptor unit of the second feature descriptor, where l d (d ab ,d′ a′b′ ;s aba′b′ ) can be expressed as: d (d,d′;s)=λ d ×s×max(0,m p -d T d′)+(1-s)×max(0,d T d′-m n );

[0090] Among them, λ d is a hyperparameter, m p With m n are positive margin and negative margin respectively, s represents the set of all correct pairs, s aba′b′ is the indicator function used to determine the description subunit d ab and d′ a′b′ Are the positions similar? Among them, pab To describe the subunit d ab The coordinate position in the third image, To describe the subunit d′ a′b′ Coordinate position in the fifth image.

[0091] The fourth loss information is used to describe the difference in feature point distribution between the second predicted feature map and the annotated feature information. P The calculation method of (X′, Y′) can refer to the calculation process of the second loss information mentioned above, which will not be repeated here.

[0092] Optionally, in some embodiments, an initial image feature extraction network is trained according to the first loss information, the second loss information, the third loss information and the fourth loss information to obtain a target image feature extraction network, including performing weighted sum processing on the first loss information, the second loss information, the third loss information and the fourth loss information to obtain target loss information, and then iteratively training the initial image feature extraction network according to the target loss information until the target loss information meets the loss condition, and using the image feature extraction network obtained by the iterative training as the target image feature extraction network.

[0093] After determining the first loss information, the second loss information, the third loss information, and the fourth loss information, the embodiment of the present disclosure may perform weighted sum processing on the first loss information, the second loss information, the third loss information, and the fourth loss information to obtain target loss information. The determination method of the target loss information may be expressed as follows:

[0094] L=L P (X,Y)+L P (X′,Y′)+λ 2 L d (D,D′,S)+λ 1 L e (E, E′);

[0095] Among them, L is the target loss information, λ 2 and λ 1 is the weight coefficient.

[0096] In the embodiment of the present disclosure, a first image corresponding to a target object captured under fill light conditions, a second image corresponding to a target object captured under no fill light conditions, and a pre-trained initial image feature extraction network are obtained, and then the color matrix and gamma value of the second image are adjusted according to the first image to obtain a third image, and then the annotation feature information corresponding to the third image is determined according to the first image, and then the third image is affine transformed to obtain a fifth image, and then the third image and the fifth image are respectively input into a feature point extraction submodel in the initial image feature extraction network to obtain a first predicted feature map corresponding to the third image and a second predicted feature map corresponding to the fifth image output by the feature point extraction submodel, and then the third image and the fifth image are respectively input into the initial image. In the feature descriptor extraction submodel in the feature extraction network, the first feature descriptor corresponding to the third image and the second feature descriptor corresponding to the fifth image output by the feature descriptor extraction submodel are obtained, and then the initial image feature extraction network is trained according to the first predicted feature map, the second predicted feature map, the annotated feature information, the first feature descriptor, the second feature descriptor, the first image and the third image to obtain the target image feature extraction network. Thus, the image feature extraction network can be jointly trained with the first image collected under fill light conditions and the second image collected under no fill light conditions, thereby effectively improving the image feature extraction ability of the image feature extraction network in complex lighting scenes, thereby effectively improving the image feature extraction effect in complex lighting scenes.

[0097] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0098] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0099] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0100] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0102] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A method for extracting features from complex illumination images using a self-supervised network, characterized in that: The method comprises: Acquire a first image corresponding to the target object captured under fill light conditions, a second image corresponding to the target object captured under no fill light conditions, and a pre-trained initial image feature extraction network; adjusting the color matrix and gamma value of the second image according to the first image to obtain a third image; Determining, according to the first image, annotation feature information corresponding to the third image; Training the initial image feature extraction network according to the first image, the third image, and the annotated feature information corresponding to the third image to obtain a target image feature extraction network; Wherein, determining the annotation feature information corresponding to the third image according to the first image includes: Performing N homography transformation processes on the first image to obtain N fourth images, wherein the value of N is a positive integer, and the processing scale of each homography transformation process is different; Acquire a first initial feature map corresponding to each of the fourth images, and a second initial feature map corresponding to the first image; Determining the annotated feature information corresponding to the third image according to the N first initial feature maps and the second initial feature maps; The determining, according to the N first initial feature maps and the second initial feature maps, the annotated feature information corresponding to the third image includes: Performing an inverse homography transformation restoration process on each of the first initial feature maps to obtain a third initial feature map; Fusing the N third initial feature maps with the second initial feature maps to obtain a fourth initial feature map; Determine the confidence level corresponding to each feature point in the fourth initial feature map; When the confidence level is greater than or equal to a confidence level threshold, a feature map composed of the feature points corresponding to at least one of the confidence levels is used as the annotated feature information.

2. The method according to claim 1, characterized in that The training of the initial image feature extraction network according to the first image, the third image and the annotated feature information corresponding to the third image to obtain a target image feature extraction network includes: Performing affine transformation processing on the third image to obtain a fifth image; The initial image feature extraction network is trained according to the first image, the third image, the fifth image and the annotated feature information to obtain a target image feature extraction network.

3. The method according to claim 2, characterized in that The training of the initial image feature extraction network according to the first image, the third image, the fifth image and the annotated feature information to obtain a target image feature extraction network includes: Inputting the third image and the fifth image into the feature point extraction submodel in the initial image feature extraction network respectively, so as to obtain a first predicted feature map corresponding to the third image and a second predicted feature map corresponding to the fifth image outputted by the feature point extraction submodel; Inputting the third image and the fifth image into the feature descriptor extraction submodel in the initial image feature extraction network respectively, so as to obtain a first feature descriptor corresponding to the third image and a second feature descriptor corresponding to the fifth image output by the feature descriptor extraction submodel; The initial image feature extraction network is trained according to the first predicted feature map, the second predicted feature map, the annotated feature information, the first feature descriptor, the second feature descriptor, the first image and the third image to obtain the target image feature extraction network.

4. The method according to claim 3, characterized in that The training of the initial image feature extraction network according to the first predicted feature map, the second predicted feature map, the annotated feature information, the first feature descriptor, the second feature descriptor, the first image, and the third image to obtain the target image feature extraction network includes: determining first loss information between the first image and the third image, wherein the first loss information is used to describe a difference in pixel intensity between the first image and the third image; Determining second loss information between the first predicted feature map and the annotated feature information, wherein the second loss information is used to describe a difference in feature point distribution between the first predicted feature map and the annotated feature information; determining third loss information between the first feature descriptor and the second feature descriptor, wherein the third loss information is used to describe a distribution difference between the first feature descriptor and the second feature descriptor; Determining fourth loss information between the second predicted feature map and the annotated feature information, wherein the fourth loss information is used to describe a difference in feature point distribution between the second predicted feature map and the annotated feature information; The initial image feature extraction network is trained according to the first loss information, the second loss information, the third loss information and the fourth loss information to obtain the target image feature extraction network.

5. The method according to claim 4, characterized in that The training of the initial image feature extraction network according to the first loss information, the second loss information, the third loss information and the fourth loss information to obtain the target image feature extraction network includes: Performing weighted sum processing on the first loss information, the second loss information, the third loss information, and the fourth loss information to obtain target loss information; The initial image feature extraction network is iteratively trained according to the target loss information until the target loss information satisfies a loss condition, and the image feature extraction network obtained by the iterative training is used as the target image feature extraction network.

6. The method according to claim 3, characterized in that The step of inputting the third image and the fifth image into the feature point extraction submodel in the initial image feature extraction network respectively to obtain a first predicted feature map corresponding to the third image and a second predicted feature map corresponding to the fifth image output by the feature point extraction submodel comprises: determining first size information of the third image and second size information of the fourth image; Inputting the third image and the fifth image into the first stacked convolution layer in the feature point extraction submodel respectively, so as to obtain a first feature map corresponding to the third image and a second feature map corresponding to the fifth image output by the first stacked convolution layer; Performing sub-pixel convolution processing on the first feature map and the second feature map respectively to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map; The size of the third feature map is adjusted according to the first size information to obtain the first predicted feature map, and the size of the fourth feature map is adjusted according to the second size information to obtain the second predicted feature map.

7. The method according to claim 3, characterized in that The step of inputting the third image and the fifth image into the feature descriptor extraction submodel in the initial image feature extraction network respectively to obtain a first feature descriptor corresponding to the third image and a second feature descriptor corresponding to the fifth image output by the feature descriptor extraction submodel comprises: Inputting the third image and the fifth image into the second stacked convolution layer in the feature descriptor extraction submodel respectively, so as to obtain a third feature descriptor corresponding to the third image and a fourth feature descriptor corresponding to the fourth image output by the second stacked convolution layer; Performing bicubic linear interpolation processing on the third feature descriptor and the fourth feature descriptor respectively to obtain a fifth feature descriptor corresponding to the third feature descriptor and a sixth feature descriptor corresponding to the fourth feature descriptor; Regularization is performed on the fifth feature descriptor to obtain the first feature descriptor, and regularization is performed on the sixth feature descriptor to obtain the second feature descriptor.

Images