An image processing method, an image processing model and an electronic device

Abstract

This application discloses an image processing method, an image processing model, and an electronic device. The method includes: obtaining a first image and a second image, wherein the first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range, the first wavelength range and the second wavelength range being different; processing the first image and the second image respectively based on the same feature extraction network to obtain a first image feature of the first image and a second image feature of the second image; performing feature transformation on the first image feature to obtain a third image feature, the third image feature being an image feature corresponding to the second wavelength range; and matching the third image feature and the second image feature to obtain an image matching result, the image matching result indicating pixels in the first image and the second image that correspond to the same object.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an image processing method, an image processing model, and an electronic device. Background Technology

[0002] Image matching is an important technique in the field of image processing. It can determine which pixels in two images match each other. Techniques such as image-based 3D reconstruction and visual positioning all rely on image matching.

[0003] Image matching encounters cross-modal scenarios, where the two images used for matching are obtained based on different lighting conditions. For example, one image might be captured by a visible light sensor using visible light, while the other is captured by an infrared sensor using infrared light. Images obtained based on different lighting conditions have different characteristics, making it difficult to obtain accurate image matching results in cross-modal scenarios. Summary of the Invention

[0004] Therefore, this application discloses the following technical solution:

[0005] The first aspect of this application provides an image processing method, comprising:

[0006] A first image and a second image are obtained. The first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range. The first wavelength range and the second wavelength range are different.

[0007] The first image and the second image are processed separately using the same feature extraction network to obtain the first image feature of the first image and the second image feature of the second image;

[0008] The first image feature is transformed to obtain a third image feature, which is an image feature corresponding to the second wavelength range;

[0009] The third image feature and the second image feature are matched to obtain an image matching result, which indicates the pixels in the first image and the second image that correspond to the same object.

[0010] Optionally, matching the third image feature and the second image feature to obtain an image matching result includes:

[0011] The third image feature and the second image feature are processed sequentially by multiple sequentially connected matching layers to obtain the layer matching result of each matching layer;

[0012] Each matching layer has input and output at a specific resolution. The input of each matching layer includes a first input feature and a second input feature with a corresponding resolution. The first input feature is obtained by processing the third image feature, and the second input feature is obtained by processing the second image feature. The layer matching result of each matching layer is obtained by adjusting the corresponding initial matching result according to the input. The initial matching result of the first matching layer is determined according to the first input feature and the second input feature. Except for the first matching layer, the initial matching result of each matching layer is obtained by processing the layer matching result of the previous matching layer.

[0013] The layer matching result of the last matching layer is obtained as the image matching result.

[0014] Optionally, the initial matching result of the matching layer is obtained by processing the layer matching result of the previous matching layer, including:

[0015] The layer matching result of the previous matching layer is upsampled to obtain an upsampled matching result with the resolution corresponding to the matching layer;

[0016] The upsampled matching result is corrected based on the first and second input features corresponding to the matching layer to obtain the initial matching result of the matching layer.

[0017] A second aspect of this application provides an image processing model, comprising:

[0018] A feature extraction network is used to process a first image and a second image to obtain a first image feature of the first image and a second image feature of the second image; the first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range, wherein the first wavelength range and the second wavelength range are different;

[0019] An adapter is used to perform feature transformation on the first image features to obtain a third image feature, wherein the third image feature is an image feature corresponding to the second wavelength range;

[0020] A matching network is used to match the third image features and the second image features to obtain an image matching result, wherein the image matching result indicates pixels in the first image and the second image that correspond to the same object.

[0021] Optionally, the matching network includes multiple sequentially connected matching layers, each with input and output at a specific resolution. The training process of the matching network includes:

[0022] Input the third sample feature obtained based on the first sample image and the second sample feature obtained based on the second sample image into the matching network to be trained;

[0023] The third loss term of the matching layer is constructed based on the output of the matching layer. The third loss term reflects the deviation between the flow field output by the matching layer and the pre-calibrated current real flow field. The resolution of the current real flow field is consistent with the output of the matching layer.

[0024] The output of the matching layer is upsampled to obtain the look-ahead flow field of the matching layer. A fourth loss term of the matching layer is constructed based on the look-ahead flow field. The fourth loss term reflects the deviation between the look-ahead flow field and the pre-calibrated look-ahead true flow field. The resolution of the look-ahead true flow field is consistent with the output of the matching layer after the matching layer.

[0025] The matching loss function is obtained based on the third and fourth loss terms of each matching layer;

[0026] With the goal of minimizing the matching loss function, the parameters of the matching network to be trained are iteratively optimized until a preset convergence condition is met, thus obtaining the trained matching network.

[0027] A third aspect of this application provides an electronic device, comprising:

[0028] Memory, used to store computer programs;

[0029] A processor for executing the computer program to perform:

[0030] A first image and a second image are obtained. The first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range. The first wavelength range and the second wavelength range are different.

[0031] The first image and the second image are processed separately using the same feature extraction network to obtain the first image feature of the first image and the second image feature of the second image;

[0032] The first image feature is transformed to obtain a third image feature, which is an image feature corresponding to the second wavelength range;

[0033] The third image feature and the second image feature are matched to obtain an image matching result, which indicates the pixels in the first image and the second image that correspond to the same object. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of an image matching result provided in an embodiment of this application;

[0037] Figure 3 This is a schematic diagram of the structure of an image processing model provided in an embodiment of this application;

[0038] Figure 4 This is a schematic diagram of the structure of a matching network provided in an embodiment of this application;

[0039] Figure 5 This is a flowchart of an adapter training method provided in an embodiment of this application;

[0040] Figure 6 This is a flowchart of a training method for a matching network provided in an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] This application provides an image processing method; please refer to [link / reference]. Figure 1 The method may include the following steps.

[0044] S101, obtain a first image and a second image. The first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range. The first wavelength range and the second wavelength range are different.

[0045] The image processing method of this embodiment can be applied to any electronic device that needs to match two images, such as robot devices of various shapes and scenarios.

[0046] The first wavelength range and the second wavelength range can be any two non-overlapping wavelength ranges of light, with no specific numerical limit. As an example, the first wavelength range can be from 0.78 micrometers (μm) to 1000 μm, meaning that the light in the first wavelength range is equivalent to infrared light (infrared radiation), and the first image is equivalent to an infrared light image obtained by an infrared sensor sensing the infrared light of an object. The second wavelength range can be from 380 nanometers (nm) to 780 nm, meaning that the light in the second wavelength range is equivalent to visible light visible to the human eye, and the second image is equivalent to a visible light image obtained by a visible light sensor sensing the reflected visible light.

[0047] S102, the first image and the second image are processed respectively based on the same feature extraction network to obtain the first image features of the first image and the second image features of the second image.

[0048] In step S102, on the one hand, the first image is input into the feature extraction network to obtain the first image features of the first image output by the feature extraction network, and on the other hand, the second image is input into the feature extraction network to obtain the second image features of the second image output by the feature extraction network.

[0049] The feature extraction network in this embodiment can be a convolutional neural network pre-trained based on multiple sample images. The feature extraction network can include multiple convolutional layers, which can sequentially calculate the corresponding image features from the images input to the feature extraction network. The structure and training methods of the feature extraction network can be found in relevant techniques in the field of image processing.

[0050] In order for the feature extraction network to correctly extract image features, each sample image used to train the feature extraction network is an image obtained by collecting light in the second wavelength range, such as an image taken based on visible light.

[0051] S103, perform feature transformation on the first image feature to obtain the third image feature, which is the image feature corresponding to the second wavelength range.

[0052] Images obtained from light of different wavelength ranges have significant differences in spectral characteristics. For example, infrared images reflect the thermal radiation of an object, while visible light images reflect the reflected light. Correspondingly, the image features obtained from images of different wavelength ranges will also have differences due to spectral characteristics. If the extracted image features are directly used for matching, the accuracy of the matching results will obviously be reduced due to the influence of these differences.

[0053] The same feature extraction network obviously cannot eliminate the differences caused by these spectral characteristics when extracting image features. Therefore, it may be necessary to transform the image features of an image within a certain wavelength range to eliminate the differences caused by these spectral characteristics.

[0054] In S103, a first image feature with spectral characteristics in a first wavelength range can be converted. While keeping other information in the first image feature (such as object outline, brightness, etc.) unchanged, the spectral characteristics in the first image feature in the first wavelength range are converted into spectral characteristics in the second wavelength range. The resulting image feature with spectral characteristics in the second wavelength range is the aforementioned third image feature.

[0055] S104, match the third image features and the second image features to obtain the image matching result, which indicates the pixels in the first image and the second image that correspond to the same object.

[0056] After obtaining the third image feature, the second and third image features can be directly matched to obtain the image matching result between the first and second images.

[0057] The image matching results obtained in this embodiment are used to characterize the pixel matching relationship between two images, that is, to reflect which pixels in the first image and which pixels in the second image match each other. Pixel matching between two images means that the pixels correspond to the same position on the surface of the object.

[0058] As an example, Figure 2 (1) and Figure 2 (2) is equivalent to the first image and the second image. The image matching result indicates that pixel 1 of the first image and pixel 1' of the second image match each other, pixel 2 of the first image and pixel 2' of the second image match each other, etc. This matching result indicates that pixel 1 and pixel 1' correspond to the same position on the surface of the object being photographed, and pixel 2 and pixel 2' correspond to another position on the surface of the object.

[0059] The image matching results obtained based on the methods and models of any embodiment of this application can be applied to any related technology that requires image matching, including but not limited to 3D reconstruction and visual positioning in the field of computer vision.

[0060] The beneficial effects of this embodiment are as follows:

[0061] Firstly, this scheme converts the first image features corresponding to the first wavelength range into the third image features corresponding to the second wavelength range, and then obtains the image matching result based on the third image features and the second image features, thus achieving image matching between the first image and the second image across the modality;

[0062] Secondly, this solution can achieve cross-modal image matching using the same feature extraction network, without the need to train multiple different feature extraction networks specifically for images with different wavelength ranges. This improves the versatility of cross-modal image processing technology and helps save the computational resources consumed in training feature extraction networks.

[0063] Third, some related technologies convert images of different wavelength ranges when processing cross-modal images, such as converting infrared images to visible light images. However, the converted images are severely distorted compared to the original images, resulting in low accuracy of the results. In contrast, this solution only converts the image features output by the feature extraction network. Compared with methods that directly convert images, this reduces the distortion caused by conversion and improves the accuracy of the results. Moreover, the amount of data for image features is generally less than that for images, thus saving computational resources.

[0064] In summary, the image processing method of this embodiment can obtain highly accurate image matching results for two images with different wavelength ranges while improving versatility and saving computing resources.

[0065] The image processing method in this embodiment can utilize Figure 3 The image processing model shown is implemented, and the image processing model includes:

[0066] The feature extraction network 301 is used to process the first image and the second image to obtain the first image features of the first image and the second image features of the second image; the first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range, wherein the first wavelength range and the second wavelength range are different;

[0067] Adapter 302 is used to perform feature transformation on the first image features to obtain the third image features, wherein the third image features are image features corresponding to the second wavelength range;

[0068] The matching network 303 is used to match the features of the third image and the features of the second image to obtain the image matching result, which indicates the pixels in the first image and the second image that correspond to the same object.

[0069] The feature extraction network, adapter, and matching network each include several parameters. The values ​​of these parameters can be determined during the training process of the feature extraction network, adapter, and matching network. The feature extraction network, adapter, and matching network can perform calculations on the input based on their own parameters to achieve the above functions.

[0070] Optionally, the method for obtaining the third image feature in step S103 may be:

[0071] Process the first image features to obtain the image feature residual of the first image features. The image feature residual represents the deviation between the image features corresponding to the first wavelength range and the image features corresponding to the second wavelength range.

[0072] The first image feature is corrected based on the image feature residual to obtain the third image feature.

[0073] The method for obtaining the third image features described above can be represented by the following formula (1).

[0074]

[0075] Where F IRt F represents the third image feature. IR The first image feature is represented by Adapter(), which represents the adapter. IR The image feature residual () represents the image feature residual obtained by the adapter after processing the first image feature. In other words, the first image feature can be input into the adapter to obtain the image feature residual output by the adapter. Then, the image feature residual and the first image feature are fused (added) to obtain the third image feature. During this process, the adapter does not directly reconstruct the image features corresponding to the second wavelength range. Instead, it calculates the offset (i.e., the image feature residual) of the image features in the first wavelength range relative to the image features in the second wavelength range through a nonlinear mapping method, and then uses the image feature residual to correct and obtain the third image feature.

[0076] The advantage of obtaining the third image feature in this way is that by calculating only the offset, the original semantics of the first image feature (such as the object outline reflected by the image feature) can be preserved to the maximum extent, avoiding direct reconstruction and obtaining image features with severe distortion, which is conducive to obtaining more accurate matching results.

[0077] In some alternative embodiments, the adapter can also directly reconstruct the input first image features, that is, generate a third image feature corresponding to the second wavelength range based on the first image features. In this case, the input of the adapter is the first image features, and the output is the third image features.

[0078] Obtaining the third image features using the above method helps reduce computation and improves the efficiency of obtaining the third image features.

[0079] Optionally, the matching network in this embodiment may include multiple matching layers, and the number of matching layers may be denoted as K, for example... Figure 4 Matching layer 1, matching layer 2 to matching layer K. The value of K can be set as needed. To improve matching efficiency, a smaller K value can be set, and to improve accuracy, a larger K value can be set. There are no restrictions. As an example, K can be equal to 3, 7 or other values.

[0080] Based on the structure of the matching network described above, matching the third image features and the second image features to obtain the image matching result can include:

[0081] The third image feature and the second image feature are processed sequentially by multiple sequentially connected matching layers to obtain the layer matching result of each matching layer;

[0082] Each matching layer has input and output at a specific resolution. The input of each matching layer includes a first input feature and a second input feature with a corresponding resolution. The first input feature is obtained by processing the third image feature, and the second input feature is obtained by processing the second image feature. The layer matching result of each matching layer is obtained by adjusting the corresponding initial matching result according to the input. The initial matching result of the first matching layer is determined according to the first input feature and the second input feature. Except for the first matching layer, the initial matching result of each matching layer is obtained by processing the layer matching result of the previous matching layer.

[0083] The image matching result is obtained based on the layer matching result of the last matching layer.

[0084] Each matching layer can calculate its initial matching result based on the correlation between its first and second input features (specifically through regression, iterative calculation, etc.) to obtain the layer matching result output by that layer. For details on how each matching layer calculates its layer matching result based on its inputs and initial matching results, please refer to relevant techniques in the field of image matching.

[0085] The layer matching result output by any matching layer (matching layer k) can be denoted as the flow field k of the corresponding layer. The initial matching result used by each matching layer can be represented by the flow field k'. For example, the output of matching layer 1 is denoted as flow field 1, and the initial matching result used for adjustment is denoted as flow field 1'. The output of matching layer 2 is flow field 2, and the initial matching result used for adjustment is denoted as flow field 2'. The output of the last matching layer (matching layer K) is flow field K, and the initial matching result used for adjustment is denoted as flow field K'. k is an integer between 1 and K, and flow field K can be used as the aforementioned image matching result. The aforementioned first matching layer can be... Figure 4 The matching layer 1 is shown.

[0086] Each matching layer k corresponds to a resolution S. k This resolution represents the resolution of the input and output of this matching layer, for example, the resolution S of matching layer 1. k The resolution is 2x2, meaning that the input features of matching layer 1 (input 1) have a resolution of 2x2 (equivalent to features containing 2x2 pixels), and the output flow field 1 also has a resolution of 2x2. From matching layer 1 to matching layer K, the corresponding resolution increases layer by layer, i.e., resolution S... kGreater than resolution S k-1 The resolution S of the matching layer K K It can be equal to the resolution of the first image, and the resolution of the first image and the second image are the same.

[0087] The resolution of each matching layer is determined by the resolutions of the second and third image features, the number of matching layers, and the resolutions of the first and second images, and remains a fixed value during image processing using the image processing model.

[0088] One method to obtain the initial matching result of the next matching layer by processing the layer matching result of the previous matching layer is to upsample the layer matching result of the previous matching layer to obtain a flow field with the resolution of the next matching layer, and use this upsampled flow field as the initial matching result of the next matching layer.

[0089] Combination Figure 4 For example, the resolution S of matching layer k-1 k-1 The output layer matching result is a flow field k-1 with a resolution of 8*8, and the resolution S of the matching layer k is 8*8. k Since the resolution is 16*16, the flow field k-1 can be upsampled to obtain a flow field with a resolution of 16*16. This upsampled flow field with a resolution of 16*16 is the flow field k' used by the matching layer k.

[0090] The flow field 1' used in matching layer 1 can be obtained by calculating the global correlation between the second and third image features. The calculation method can be found in the relevant techniques in the field of image matching, which will not be elaborated here.

[0091] The input 1 of the matching layer 1 may include a first input feature 1 and a second input feature 1. The first input feature 1 may be a third image feature, or it may be an image feature with a resolution S1 obtained by upsampling or downsampling the third image feature. The second input feature 1 may be a second image feature, or it may be an image feature with a resolution S1 obtained by upsampling or downsampling the second image feature.

[0092] For any value k greater than or equal to 2 and less than or equal to K, the input k of the matching layer k can be obtained by upsampling the input k-1 of its preceding matching layer k-1. That is, the input k of the matching layer k includes a first input feature k and a second input feature k. The first input feature k is obtained by upsampling the first input feature k-1 and has a resolution S. k The second input feature k is obtained by upsampling the second input feature k-1, and has a resolution S. k .

[0093] For any matching layer k, its output flow field k contains the corresponding resolution S k The number of motion vectors, such as S k If the flow field k is 8*8, then the flow field k contains 64 (8*8) motion vectors. These motion vectors correspond one-to-one with the pixels of the first input feature (or the second input feature) of the input matching layer k. Each motion vector represents the displacement between the corresponding pixels of the first input feature and the second input feature.

[0094] As an example, the resolution S1 of the matching layer 1 is 2*2, and its output flow field 1 can be represented by Table 1, where each cell contains one motion vector.

[0095] Table 1

[0096] （1，2） （2，2） （1，1） （2，1）

[0097] The method for obtaining image matching results based on the layer matching results of the last matching layer can include any of the following:

[0098] The layer matching result of the last matching layer is directly used as the image matching result;

[0099] After upsampling or downsampling the layer matching result of the last matching layer, an image matching result with the same resolution as the first image is obtained.

[0100] The advantage of obtaining image matching results using the above method is that:

[0101] By iteratively processing multiple matching layers, the accuracy of the flow field output by each matching layer can be continuously improved, ultimately resulting in more accurate image matching results.

[0102] Optionally, the method for processing the layer matching results of the previous matching layer to obtain the initial matching result of the matching layer can be:

[0103] The matching result of the previous matching layer is upsampled to obtain an upsampled matching result with the resolution corresponding to the matching layer.

[0104] The upsampling matching result is corrected based on the first and second input features corresponding to the matching layer to obtain the initial matching result of the matching layer.

[0105] The following example, using matching layer k (k greater than or equal to 2), illustrates the method for obtaining the initial matching result.

[0106] Obtain the flow field k-1 output from the previous matching layer (i.e., matching layer k-1), upsample the flow field k-1, and obtain a flow field with resolution S. k The upsampled matching result (i.e., the resolution corresponding to matching layer k);

[0107] Global correlation calculation is performed on the first input feature k and the second input feature k corresponding to the matching layer k to obtain the reference flow field corresponding to the matching layer k. The calculation method can refer to the method of calculating the global correlation of the second image feature and the third image feature to obtain flow field 1'. This reference flow field reflects the pixel matching relationship between the first input feature k and the second input feature k obtained in the preliminary analysis. The resolution of this reference flow field is S. k ;

[0108] The reference flow field and the upsampled matching result are weighted and summed according to a preset weight. The result is used as the initial matching result of the matching layer k, i.e., the aforementioned flow field k'.

[0109] The advantage of obtaining the initial matching results using the above method is that:

[0110] The above method can introduce the correlation between the first and second input features into the initial matching result. This correlation can be used as a reference when the initial matching result is adjusted in the matching layer k, which helps to improve the accuracy of the flow field k output by the matching layer k, and thus improves the accuracy of the final image matching result.

[0111] The image processing model used in this embodiment can be pre-trained using several samples, and the image processing model can be trained in stages. For example, in the first stage, the feature extraction network and the matching network can be trained first. After the feature extraction network and the matching network are trained, in the second stage, the parameters of the feature extraction network and the matching network are kept unchanged, and the adapter is trained separately. Alternatively, the image processing model can be trained in stages. Figure 3 The three modules are trained sequentially: first, the feature extraction network is trained; then, the matching network is trained; and finally, the adapter is trained. The parameters of the other modules remain unchanged when any one module is trained.

[0112] Among them, see Figure 5 The adapter training process may include the following steps.

[0113] S501, Input the first sample image and the second sample image into the feature extraction network to obtain the first sample feature of the first sample image and the second sample feature of the second sample image; The first sample image is obtained by collecting light in the first wavelength range and the second sample image is obtained by collecting light in the second wavelength range.

[0114] S502, input the first sample features into the adapter to be trained to obtain the third sample features.

[0115] S503, constructs a first loss term based on the third sample features and the second sample features. The first loss term reflects the deviation between the transformed features and the true features. The transformed features refer to the features obtained by the adapter transformation, and the true features refer to the features extracted by the feature extraction network from the image in the second wavelength range.

[0116] S504. A second loss term is constructed based on the output of the matching network. The second loss term reflects the deviation between the output of the matching network and the pre-calibrated true matching result.

[0117] S505, add the second loss term to the weighted first loss term to obtain the total loss function.

[0118] S506 aims to minimize the total loss function by iteratively optimizing the parameters of the adapter to be trained until the preset convergence condition is met, thus obtaining the trained adapter.

[0119] Before training the adapter, you can first obtain the trained feature extraction network and matching network, and then use the feature extraction network and matching network to carry out the training process of the adapter.

[0120] In step S501, the first and second sample images can be sample images acquired from the same scene. For example, a visible light camera and an infrared camera can be used to simultaneously capture images of the same scene, with the image captured by the infrared camera serving as the first sample image and the image captured by the visible light camera serving as the second sample image.

[0121] The feature extraction network processes the first sample image to obtain the first sample feature, and processes the second sample image to obtain the second sample feature. The method by which the adapter to be trained in S502 processes the first sample feature to obtain the third sample feature is described above in the section on converting first image features into third image features.

[0122] The true features in step S503 refer to the features obtained after processing an image corresponding to the second wavelength range and the same scene as the first sample image through a feature extraction network. Specifically, if the first and second sample images can be sample images acquired from the same scene, then in S503, the third sample feature output by the adapter can be considered the transformed feature, the second sample feature can be considered the true feature, and a first loss term is constructed based on the deviation between the third and second sample features.

[0123] The role of the first loss term is to reflect the degree of difference between the features converted by the adapter and the real image features corresponding to the second wavelength range. By introducing the first loss term, the adapter can be guided to learn the inherent spectral characteristics of the real image features in the second wavelength range during the training process, so that the trained adapter can convert the features of the first wavelength range to the second wavelength range to the maximum extent.

[0124] The true matching result of step S504 can be obtained based on the first sample image and the second sample image. It can be obtained by manual annotation or by processing the two sample images using other image matching techniques with higher accuracy.

[0125] In step S504, the third sample features and the second sample features can be input into the matching network to obtain the sample matching result output by the matching network (i.e., the result output by the matching network). The second loss term is constructed based on the deviation between the sample matching result and the true matching result.

[0126] Both the sample matching result and the true matching result can be flow fields containing multiple motion vectors, and the two matching results have the same resolution. These motion vectors represent the positional relationship of the pixels that are matched between the first sample image and the second sample image. The method for constructing the second loss term can be to calculate the difference between the motion vectors corresponding to the same position between the sample matching result and the true matching result, calculate the average of all differences, or calculate the average of the squares of all differences, and use the result as the second loss term.

[0127] In step S505, the preset first weighting coefficient Lam can be used as a basis. a The second loss term and the first loss term are weighted and summed to obtain the total loss function, which can be expressed by the following formula (2).

[0128]

[0129] Where L Total Let L represent the total loss function. task L represents the second loss term. align Lam represents the first loss term and the first weighting coefficient. a It can be a real number greater than 0 and less than 1, and its specific value can be set as needed.

[0130] In step S506, the value of the total loss function can be calculated. If this value does not meet the convergence condition (e.g., not less than a preset convergence threshold), the values ​​of each parameter in the adapter to be trained are updated based on the total loss function. After the update, the value of the total loss function is recalculated using the method described above. This process is repeated until the value meets the convergence condition. The method for updating the parameters in the adapter based on the total loss function can be found in relevant techniques in the fields of neural networks and deep learning, and will not be elaborated upon here.

[0131] Optionally, a first loss term is constructed based on the features of the third sample and the features of the second sample, including:

[0132] Calculate the error between the corresponding positions of the third sample feature and the second sample feature;

[0133] The mean of the squared values ​​of multiple errors is used to obtain the first loss term.

[0134] The method for constructing the first loss term described above can be represented by the following formula (3).

[0135]

[0136] Among them, F RGB F represents the second sample feature. RGB (p) represents the data at position p in the second sample feature (equivalent to a pixel in the image), F' IR F' represents the third sample feature. IR (p) represents the pixel at position p in the third sample feature, P0 represents the set of all positions in the second sample feature, and N is the total number of data in the second sample feature. For example, if the resolution of the second sample feature is 8*8, then N equals 64, and P0 is the set of these 64 positions.

[0137] The method described above for obtaining the first loss term is equivalent to using the mean squared error (MSE) between the third sample feature and the second sample feature as the first loss term. Optionally, the L2 norm between the third sample feature and the second sample feature can also be calculated as the first loss term; the calculation method for the L2 norm can be found in related techniques.

[0138] Optionally, the matching network comprises multiple sequentially connected matching layers, each with inputs and outputs at a specific resolution. See [link to relevant documentation]. Figure 6 The training process of the matching network can include the following steps.

[0139] S601, input the third sample feature obtained based on the first sample image and the second sample feature obtained based on the second sample image into the matching network to be trained.

[0140] S602, construct the third loss term of the matching layer based on the output of the matching layer. The third loss term reflects the deviation between the flow field output by the matching layer and the pre-calibrated current real flow field. The resolution of the current real flow field is consistent with the output of the matching layer.

[0141] S603, the output of the matching layer is upsampled to obtain the look-ahead flow field of the matching layer. Based on the look-ahead flow field, the fourth loss term of the matching layer is constructed. The fourth loss term reflects the deviation between the look-ahead flow field and the pre-calibrated look-ahead real flow field. The resolution of the look-ahead real flow field is consistent with the output of the next matching layer of the matching layer.

[0142] S604, the matching loss function is obtained based on the third and fourth loss terms of each matching layer.

[0143] S605 aims to minimize the matching loss function by iteratively optimizing the parameters of the matching network to be trained until the preset convergence condition is met, thus obtaining the trained matching network.

[0144] In this embodiment, the structure and working principle of the matching network to be trained can be found in [reference needed]. Figure 4 The corresponding embodiments are not described in detail.

[0145] The methods for obtaining the second and third sample features can be found in S501 and S502.

[0146] The following example, using the k-th matching layer of the matching network, illustrates how the third and fourth loss terms of the matching layers in S602 and S603 are obtained, where k is greater than or equal to 1 and less than the total number of matching layers K.

[0147] Before training using the method of this embodiment, K real flow fields corresponding one-to-one with the K matching layers are obtained in advance through manual calibration or other methods. The real flow fields can be represented by GT, and the K real flow fields can be denoted as GT. k , where k is an integer from 1 to K. The resolution of each real flow field is consistent with the resolution of the flow field output by its corresponding matching layer, i.e., GT. k The resolution is S k (The resolution of the flow field output by the matching layer k).

[0148] In S602 and S603, the matching network to be trained processes the second and third sample features. The processing procedure is consistent with the method for processing the second and third image features in the previous embodiments. During the processing, the flow field output by the matching layer k is denoted as the sample flow field, and W is used. k express.

[0149] For the matching layer k, the sample flow field W output by it can be used as a basis. k and the corresponding real flow field GT k The difference between them constructs the third loss term of the matching layer k, which can be denoted as L. cur (k), this loss term reflects the deviation between the flow field output by the k-th matching layer in the matching network and the real flow field (also called robust regression error). This process can be expressed by the following formula (4). For matching layer k, the real flow field GT k It refers to the current real flow field pre-calibrated in S602.

[0150]

[0151] In formula (4), CurLoss() represents the calculation of W. k and GT kThe deviation between the two values ​​yields a function of the third loss term, the form and calculation process of which are not limited. For example, this function can calculate W. k and GT k The distance between motion vectors at the same position between them can be used as the average of multiple distances, the mean square error, or other statistical values ​​as the third loss term of the matching layer k. Alternatively, a function for calculating loss from other related techniques in the field of image matching can be used to calculate the third loss term.

[0152] For the matching layer k, its output sample flow field W can be... k Upsampling is performed to obtain the corresponding look-ahead flow field k, denoted as UpW. k UpW forward flow field k The resolution of the matching layer k is the same as the output of the matching layer after matching layer k, i.e., UpW k The resolution is equal to the resolution S of the flow field output by the matching layer k+1. k+1 It is also equal to the true flow field GT corresponding to the matching layer k+1. k+1 The resolution.

[0153] Based on this, UpW can be calculated. k and GT k+1 The deviation between them is used to construct the fourth loss term of the matching layer k, which can be denoted as L. next (k). Wherein, for the matching layer k, the real flow field GT k+1 This is equivalent to the pre-calibrated look-ahead real flow field in S603. This calculation process can be expressed by formula (5).

[0154]

[0155] The advantage of constructing a fourth loss term is that, during training, the fourth loss term can be used to pre-simulate the transmission error generated when the flow field of each matching layer is transmitted to the next matching layer. By minimizing the matching loss function (including the fourth loss term), the matching network is constrained to continuously reduce this transmission error during training, thereby further improving the accuracy of image matching results.

[0156] Optionally, based on the above method for calculating the fourth loss term, it can be seen that the fourth loss term of matching layer k requires the actual flow field corresponding to the next matching layer (matching layer k+1). For the last matching layer (matching layer K) in the matching network, since it has no next matching layer, the fourth loss term corresponding to matching layer K can be defined as 0, i.e., L. next (K) = 0.

[0157] In S604, the fourth loss term of each matching layer can be based on the second weighting coefficient Lam. bAfter weighting, it is added to the third loss term of each matching layer, and the third loss terms of all matching layers in the matching network and the weighted fourth loss term are fused to form the matching loss function. This process can be expressed by the following formula (6). Where L all This represents the matching loss function of S604. The second weighting coefficient is a real number greater than 0 and less than 1, and its value can be set as needed, for example, equal to 0.25, without limitation.

[0158]

[0159] In S605, the parameters of the matching network to be trained are iteratively optimized. This can be done by iteratively optimizing only the parameters of the matching network while keeping the parameters of the feature extraction network unchanged, or by iteratively optimizing the parameters of both the matching network and the feature extraction network simultaneously, which means training both networks at the same time.

[0160] For methods to iteratively optimize the parameters of the matching network (or the matching network and the feature extraction network) to be trained, please refer to the previous section on methods for iteratively optimizing the parameters of the adapter to be trained.

[0161] In some alternative embodiments, the matching loss function L of S604 can be used during adapter training. all Instead of the aforementioned second loss term, the total loss function can be trained by adding the matching loss function and the weighted first loss term; or, a matching loss function can be further introduced during adapter training, meaning the total loss function can be trained by adding the second loss term, the matching loss function, and the weighted first loss term.

[0162] Optionally, a fourth loss term for the matching layer is constructed based on the look-ahead flow field of the matching layer, including:

[0163] Calculate the distance between the look-ahead flow location containing motion vectors and the look-ahead actual flow location containing motion vectors;

[0164] The average of multiple distances is used as the fourth loss term in the matching layer.

[0165]

[0166] The above construction method can be expressed by formula (7). Where P1 represents the look-ahead flow field UpW k The set of all positions in UpW k (p) represents the motion vector at position p in the look-through flow field, GT k+1 (p) represents the motion vector at position p in the look-ahead real flow field, Dis() represents the calculation of the distance between the two motion vectors within the parentheses, and the method for calculating the distance between vectors can be found in related technologies. N is the look-ahead flow field UpW. k The total number of motion vectors contained therein. For example, the look-through flow field UpW.k If the resolution is 8*8, then N equals 64.

[0167] In some optional embodiments, the mean square error of multiple distances or other statistical indicators can also be used as the fourth loss term of the corresponding matching layer, without limitation.

[0168] Optionally, W can also be calculated. k and GT k The L1 norm or End-Point Error (EPE) between the two points is used as the third loss term in the matching layer k to calculate UpW. k and GT k+1 The L1 norm or EPE between the two is used as the fourth loss term in the matching layer k. The calculation methods of L1 norm and EPE can be found in the relevant techniques in the field of image matching, and will not be elaborated here.

[0169] The upsampling processing involved in any embodiment of this application can be implemented by a bilinear interpolation algorithm, or by other conventional upsampling methods in related technologies.

[0170] This application also provides an electronic device, please refer to [link to application]. Figure 7 The electronic device may include a memory 701 and a processor 702.

[0171] Memory 701 is used to store computer programs;

[0172] Processor 702 is used to execute computer programs to perform:

[0173] A first image and a second image are obtained. The first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range. The first wavelength range and the second wavelength range are different.

[0174] The first image and the second image are processed separately using the same feature extraction network to obtain the first image features of the first image and the second image features of the second image.

[0175] The first image features are transformed to obtain the third image features, which are image features corresponding to the second wavelength range.

[0176] The third image features and the second image features are matched to obtain the image matching result, which indicates the pixels in the first image and the second image that correspond to the same object.

[0177] The above electronic devices can be robotic devices of any shape and applicable to any scenario.

[0178] The working principle of the electronic device in this embodiment can be found in the relevant steps of the image processing method and the aforementioned image processing model in the previous embodiment, and will not be repeated here.

[0179] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0180] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, comprising: A first image and a second image are obtained. The first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range. The first wavelength range and the second wavelength range are different. The first image and the second image are processed separately using the same feature extraction network to obtain the first image feature of the first image and the second image feature of the second image; The first image feature is transformed to obtain a third image feature, which is an image feature corresponding to the second wavelength range; The third image feature and the second image feature are matched to obtain an image matching result, which indicates the pixels in the first image and the second image that correspond to the same object.

2. The method according to claim 1, wherein performing feature transformation on the first image features to obtain the third image features includes: The first image feature is processed to obtain the image feature residual of the first image feature, wherein the image feature residual represents the deviation between the image feature corresponding to the first wavelength range and the image feature corresponding to the second wavelength range; The first image feature is corrected based on the image feature residual to obtain the third image feature.

3. The method according to claim 1, wherein matching the third image feature and the second image feature to obtain an image matching result includes: The third image feature and the second image feature are processed sequentially by multiple sequentially connected matching layers to obtain the layer matching result of each matching layer; Each matching layer has input and output at a specific resolution. The input of each matching layer includes a first input feature and a second input feature with a corresponding resolution. The first input feature is obtained by processing the third image feature, and the second input feature is obtained by processing the second image feature. The layer matching result of each matching layer is obtained by adjusting the corresponding initial matching result according to the input. The initial matching result of the first matching layer is determined according to the first input feature and the second input feature. Except for the first matching layer, the initial matching result of each matching layer is obtained by processing the layer matching result of the previous matching layer. The image matching result is obtained based on the layer matching result of the last matching layer.

4. The method according to claim 3, processing the layer matching result of the previous matching layer to obtain the initial matching result of the matching layer, includes: The layer matching result of the previous matching layer is upsampled to obtain an upsampled matching result with the resolution corresponding to the matching layer; The upsampled matching result is corrected based on the first and second input features corresponding to the matching layer to obtain the initial matching result of the matching layer.

5. An image processing model, comprising: A feature extraction network is used to process a first image and a second image to obtain a first image feature of the first image and a second image feature of the second image. The first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range, wherein the first wavelength range and the second wavelength range are different. An adapter is used to perform feature transformation on the first image features to obtain a third image feature, wherein the third image feature is an image feature corresponding to the second wavelength range; A matching network is used to match the third image features and the second image features to obtain an image matching result, wherein the image matching result indicates pixels in the first image and the second image that correspond to the same object.

6. The model according to claim 5, wherein the training process of the adapter includes: The first sample image and the second sample image are input into the feature extraction network to obtain the first sample feature of the first sample image and the second sample feature of the second sample image; the first sample image is obtained by collecting light within the first wavelength range, and the second sample image is obtained by collecting light within the second wavelength range. Input the first sample features into the adapter to be trained to obtain the third sample features; A first loss term is constructed based on the third sample features and the second sample features. The first loss term reflects the deviation between the transformed features and the true features. The transformed features refer to the features obtained by the adapter transformation, and the true features refer to the features extracted by the feature extraction network from the image in the second wavelength range. A second loss term is constructed based on the output of the matching network. The second loss term reflects the deviation between the output of the matching network and the pre-calibrated true matching result. Add the second loss term to the weighted first loss term to obtain the total loss function; With the goal of minimizing the total loss function, the parameters of the adapter to be trained are iteratively optimized until a preset convergence condition is met, thus obtaining a trained adapter.

7. The model according to claim 6, wherein constructing the first loss term based on the third sample features and the second sample features includes: Calculate the error of the data at corresponding positions between the third sample feature and the second sample feature; The squared values ​​of the multiple errors are used to calculate the mean, resulting in the first loss term.

8. The model according to claim 5, wherein the matching network comprises a plurality of sequentially connected matching layers, each matching layer having input and output at a specific resolution, and the training process of the matching network includes: Input the third sample feature obtained based on the first sample image and the second sample feature obtained based on the second sample image into the matching network to be trained; The third loss term of the matching layer is constructed based on the output of the matching layer. The third loss term reflects the deviation between the flow field output by the matching layer and the pre-calibrated current real flow field. The resolution of the current real flow field is consistent with the output of the matching layer. The output of the matching layer is upsampled to obtain the look-ahead flow field of the matching layer. A fourth loss term of the matching layer is constructed based on the look-ahead flow field. The fourth loss term reflects the deviation between the look-ahead flow field and the pre-calibrated look-ahead true flow field. The resolution of the look-ahead true flow field is consistent with the output of the matching layer after the matching layer. The matching loss function is obtained based on the third and fourth loss terms of each matching layer; With the goal of minimizing the matching loss function, the parameters of the matching network to be trained are iteratively optimized until a preset convergence condition is met, thus obtaining the trained matching network.

9. The model according to claim 8, wherein constructing the fourth loss term of the matching layer based on the look-ahead flow field of the matching layer comprises: Calculate the distance between the look-ahead flow location containing motion vectors and the look-ahead actual flow location containing motion vectors; The average of the multiple distances is used as the fourth loss term of the matching layer.

10. An electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to perform: A first image and a second image are obtained. The first image is obtained by collecting light within a first wavelength range, and the second image is obtained by collecting light within a second wavelength range. The first wavelength range and the second wavelength range are different. The first image and the second image are processed separately using the same feature extraction network to obtain the first image feature of the first image and the second image feature of the second image; The first image feature is transformed to obtain a third image feature, which is an image feature corresponding to the second wavelength range; The third image feature and the second image feature are matched to obtain an image matching result, which indicates the pixels in the first image and the second image that correspond to the same object.

Images