Image processing model training method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a method and device for training an image processing model, the image processing model comprising a visual transformation network and a category weight network, the method comprising: inputting at least one cropped sub-image corresponding to an enhanced sample image into the visual transformation network, and obtaining encoded sub-images corresponding to the at least one cropped sub-image respectively; inputting any one of the encoded sub-images, a cropped sub-image corresponding to the encoded sub-image, and an encoded image obtained by splicing all the encoded sub-images into the category weight network, and obtaining a de-enhanced sample image corresponding to the enhanced sample image; and adjusting parameters of the visual transformation network and the category weight network based on a standard sample image corresponding to the enhanced sample image and the de-enhanced sample image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a training method, apparatus, electronic device, and storage medium for an image processing model. Background Technology

[0002] In the field of autonomous driving or robot localization, vehicle or robot visual localization is usually based on high-precision visual feature maps, which includes two stages: mapping and localization. In the mapping stage, a high-precision visual feature map needs to be generated. In the localization stage, feature points in real-time acquired images need to be extracted and then matched with the high-precision visual feature map to obtain the real-time pose of the vehicle or robot. However, when the real-time acquired images differ significantly from the high-precision visual feature map due to factors such as brightness and clarity, it is difficult to achieve matching between the real-time acquired images and the high-precision visual feature map, resulting in vehicle or robot localization failure. Summary of the Invention

[0003] This disclosure provides a training method, apparatus, electronic device, and storage medium for an image processing model, to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this disclosure, a method for training an image processing model is provided, the model comprising a visual transformation network and a category weight network, the method comprising:

[0005] At least one cropped sub-image corresponding to the enhanced sample image is input into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image;

[0006] Input any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by stitching together all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image;

[0007] The parameters of the visual transformation network and the category weight network are adjusted based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image.

[0008] In the above scheme, before inputting at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image, the method further includes:

[0009] Preprocess the original sample images to obtain standard sample images;

[0010] The standard sample image is enhanced to obtain an enhanced sample image;

[0011] The enhanced sample image is cropped to obtain at least one cropped sub-image corresponding to the enhanced sample image.

[0012] In the above scheme, the step of inputting at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image includes:

[0013] The at least one cropped sub-image is input into the linear layer of the visual transformation network, and the output of the linear layer is confirmed to be the linear projection corresponding to each cropped sub-image.

[0014] The linear projection corresponding to each cropped sub-image and the classification identifier of each cropped sub-image in the enhanced sample image are input into the encoder included in the visual transformation network, and the output of the encoder is confirmed to be the coded sub-image corresponding to each of the at least one cropped sub-image.

[0015] In the above scheme, the step of inputting any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by concatenating all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image includes performing the following processing on each coded sub-image and the cropped sub-image corresponding to each coded sub-image:

[0016] The encoded sub-image is input into the first neural network layer of the category weight network to obtain the first encoded feature corresponding to the encoded sub-image;

[0017] The cropped sub-image corresponding to the encoded sub-image is input into the second neural network layer of the category weight network to obtain the second encoded feature corresponding to the cropped image;

[0018] The encoded image is input into the third neural network layer of the category weight network to obtain the third encoded feature corresponding to the encoded image.

[0019] In the above scheme, the step of inputting any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by concatenating all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image includes performing the following processing on each coded sub-image and the cropped sub-image corresponding to each coded sub-image:

[0020] Based on the first coding feature, the autocorrelation strength of the first coding feature is obtained;

[0021] The third encoded feature is masked to obtain the first matrix;

[0022] The autocorrelation intensity is divided into a non-masked region and a masked region based on the first matrix.

[0023] In the above scheme, the step of inputting any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by concatenating all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image includes performing the following processing on each coded sub-image and the cropped sub-image corresponding to each coded sub-image:

[0024] Based on the pixel weighting weights of the non-masked region and the second encoding feature, the pixel weighting output of the non-masked region is confirmed;

[0025] Based on the pixel weighting of the masked region and the first encoding feature, the pixel weighting output of the masked region is confirmed;

[0026] Based on the unmasked region and the masked region, the first and second combining coefficients are determined;

[0027] Based on the first concatenation coefficient, the second concatenation coefficient, the pixel weighted output of the non-masked region, and the pixel weighted output of the masked region, the inverse enhancement coded sub-image corresponding to the cropped sub-image is confirmed;

[0028] The stitching result of the inverse-enhanced coded sub-image corresponding to each cropped sub-image is confirmed as the inverse-enhanced coded sample image;

[0029] The anti-enhanced sample image is obtained by performing convolutional decoding on the anti-enhanced sample image.

[0030] In the above scheme, adjusting the parameters of the visual transformation network and the category weight network based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image includes:

[0031] Based on the value of each pixel in the standard sample image and the value of each pixel in the anti-enhanced sample image, the loss function of the image processing model is determined;

[0032] Based on the loss function of the image processing model, the parameters of the visual transformation network and the category weight network are adjusted.

[0033] In the above scheme, determining the loss function of the image processing model based on the value of each pixel in the standard sample image and the value of each pixel in the anti-enhanced sample image includes:

[0034] Based on the value of each pixel in the standard sample image, the value of each pixel in the anti-enhanced sample image, and the L1 loss function, the reconstruction loss sub-function is determined;

[0035] Based on the value of each pixel in the standard sample image, the value of each pixel in the anti-enhanced sample image, and the visualized activation heatmap, the perceptual loss sub-function is confirmed;

[0036] Based on the value of each pixel in the anti-enhanced sample image, the generator loss function is determined;

[0037] Based on the reconstruction loss function, the perception loss function, and the generator loss function, the loss function of the image processing model is determined.

[0038] According to a second aspect of this disclosure, a localization method is provided, implemented based on an image processing model obtained by the training method of the image processing model described in the first aspect above, the method comprising:

[0039] Based on the image processing model, the real-time image and the constructed image are normalized respectively to obtain the real-time features corresponding to the real-time image and the constructed features corresponding to the constructed image.

[0040] The Global Navigation Satellite System (GNSS) corresponding to the real-time image is fused with the mapping features, and the feature map is confirmed based on the result of the feature fusion.

[0041] Based on the feature map and the real-time features, the location information of the device that acquired the real-time image is confirmed.

[0042] According to a third aspect of this disclosure, a training apparatus for an image processing model is provided, the image processing model including a visual transformation network and a category weight network, the apparatus comprising:

[0043] The first training unit is used to input at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image respectively;

[0044] The second training unit is used to input any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by stitching together all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image;

[0045] The adjustment unit is used to adjust the parameters of the visual transformation network and the category weight network based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image.

[0046] According to a fourth aspect of this disclosure, a positioning device is provided, implemented based on an image processing model obtained by the training method of the image processing model described in the first aspect above, the device comprising:

[0047] The first training unit is used to input at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image respectively;

[0048] The second training unit is used to input any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by stitching together all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image;

[0049] The adjustment unit is used to adjust the parameters of the visual transformation network and the category weight network based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image.

[0050] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0051] At least one processor; and

[0052] A memory communicatively connected to the at least one processor; wherein,

[0053] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0054] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0055] The training method of the image processing model disclosed herein involves inputting at least one cropped sub-image corresponding to an enhanced sample image into the visual transformation network to obtain coded sub-images corresponding to each of the at least one cropped sub-image; inputting any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by stitching together all the coded sub-images into a class weight network to obtain an anti-enhanced sample image corresponding to the enhanced sample image; and adjusting the parameters of the visual transformation network and the class weight network based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image. This model can reduce the differences between images acquired under different environmental conditions, making the images more consistent, improving the matching success rate between images and high-precision visual feature maps, and enhancing the localization effect of vehicles or robots.

[0056] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0057] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0058] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0059] Figure 1 A schematic diagram of an optional process for training an image processing model provided in an embodiment of this disclosure is shown.

[0060] Figure 2 An optional schematic diagram of the training process of the image processing model provided in an embodiment of this disclosure is shown;

[0061] Figure 3 A schematic diagram of an optional structure of the image processing model provided in an embodiment of this disclosure is shown;

[0062] Figure 4 A schematic diagram of an optional structure of the category weight network provided in an embodiment of this disclosure is shown;

[0063] Figure 5 A schematic diagram of another optional structure of the category weight network provided in this disclosure embodiment is shown;

[0064] Figure 6 This diagram illustrates the effect of image processing based on the image processing model provided in the embodiments of this disclosure.

[0065] Figure 7 This illustration shows an optional flowchart of the positioning method provided in an embodiment of the present disclosure;

[0066] Figure 8 A schematic diagram of the positioning method provided in an embodiment of this disclosure is shown;

[0067] Figure 9 A schematic diagram of an optional structure of a training apparatus for an image processing model provided in an embodiment of this disclosure is shown;

[0068] Figure 10 A schematic diagram of an optional structure of the positioning device provided in an embodiment of this disclosure is shown;

[0069] Figure 11A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0070] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0071] In the current field of autonomous driving or robot localization, visual localization is typically based on existing high-precision visual feature maps. It mainly includes two stages: mapping and localization. In the mapping stage, data is simultaneously acquired from GNSS, Inertial Measurement Unit (IMU), LiDAR, and camera. The images are then preprocessed, such as by correcting distortion and resizing. Feature points are extracted from the images, and the vehicle's pose is calculated. Finally, a high-precision visual feature map corresponding to the GNSS coordinates is formed through time alignment. In the localization stage, the real-time acquired images are first preprocessed, then image feature points are extracted. These are then compared with the high-precision visual feature map obtained during the mapping process to determine the real-time pose of the vehicle or robot.

[0072] When the parameters such as brightness / clarity of the image acquired during real-time positioning differ significantly from those of the image acquired during the high-precision visual feature map construction process, it becomes difficult to match the real-time image with the pre-built map, leading to positioning failure.

[0073] In view of the deficiencies in the related technologies, this disclosure provides a training method for an image processing model and a localization method based on the image processing model, which can solve some or all of the above-mentioned technical problems.

[0074] Figure 1 This illustration shows an optional flowchart of a training method for an image processing model provided in an embodiment of this disclosure. Figure 2 An optional schematic diagram of the training process of the image processing model provided in an embodiment of this disclosure is shown. Figure 3 A schematic diagram of an optional structure of the image processing model provided in an embodiment of this disclosure is shown.

[0075] like Figure 2As shown, the training device for the image processing model (hereinafter referred to as the first device) preprocesses the acquired original sample images to obtain standard sample images. The preprocessing may include at least one of the following: histogram normalization (ESRGan) processing, adjusting the size of the original image, and correcting distortion. Then, the standard sample images are enhanced to obtain enhanced sample images, where the enhancement model can use SurfelGAN. The purpose of the enhancement processing is to adjust the brightness, sharpness, and add occlusions to the standard sample images, making them closer to naturally acquired real-world images. The image processing model is then trained using the enhanced sample images, and anti-enhanced sample images are obtained (i.e., the enhancement model's processing result has been removed, making them closer to or the same as the standard sample images). The image processing model is then trained again based on either the standard sample images or the anti-enhanced sample images. The specific training process of the image processing model is as follows:

[0076] Step S101: Input at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image.

[0077] In some embodiments, after acquiring the enhanced sample image, the first device crops the enhanced sample image to obtain at least one cropped sub-image corresponding to the enhanced sample image. Each cropped sub-image is the same size, and all cropped sub-images can be stitched together to form the enhanced sample image.

[0078] like Figure 3 As shown, assuming the augmented sample image (Image) is 720×720 in size, then each cropped sub-image (Patch) can be 16×16 in size, meaning the augmented sample image is divided into 2025 cropped sub-images of equal size. It is important to understand that... Figure 3 This is for illustrative purposes only, and not all cropped sub-images are shown.

[0079] In some embodiments, the first device inputs the at least one cropped sub-image into the visual transformation network to obtain coded sub-images corresponding to the at least one cropped sub-image.

[0080] The structure of the visual transformation network can be as follows: Figure 3 The image shows a Standard Vision Transformer (Standard VIT) network, which includes a linear layer and an encoder.

[0081] In a specific implementation, the first device inputs the at least one cropped sub-image into the linear layer included in the visual transformation network, and confirms that the output of the linear layer is the linear projection corresponding to each cropped sub-image;

[0082] In specific implementation, the first device inputs each cropped sub-image into the linear layer, performs linear projection on each cropped sub-image, and obtains the linear projection corresponding to each cropped sub-image. Then, the linear projection corresponding to each cropped sub-image and the classification token (cls token) of each cropped sub-image in the enhanced sample image are input into the encoder included in the visual transformation network, and the output of the encoder is confirmed to be the encoded sub-image corresponding to each of the at least one cropped sub-image. The classification token may include the position embedding value of each cropped sub-image, i.e. Figure 3 The diagram shows an ellipse containing the numbers 1, 2, 3, ... 7. Furthermore, the encoder's input can also include additional patches and class tokens, i.e. Figure 3 A schematic diagram of an ellipse with a center of 0.

[0083] Step S102: Input any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by stitching together all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image.

[0084] In some embodiments, the input to the Class Guided Aware Network includes any encoded sub-image, a cropped sub-image corresponding to the any encoded sub-image, and an encoded image obtained by stitching together all the encoded sub-images.

[0085] Figure 4 A schematic diagram of an optional structure of the category weight network provided in an embodiment of this disclosure is shown.

[0086] like Figure 4 As shown, the category weight network includes a first neural network layer, a third neural network layer, a SuftGate, a classification encoder layer, and a convolutional decoder. In some alternative embodiments, the weight category network may further include a second neural network layer.

[0087] In specific implementation, the first device inputs the encoded sub-image into the first neural network layer of the category weight network to obtain the first encoded feature corresponding to the encoded sub-image; inputs the cropped sub-image corresponding to the encoded sub-image into the second neural network layer of the category weight network to obtain the second encoded feature corresponding to the cropped sub-image; and inputs the encoded image into the third neural network layer of the category weight network to obtain the third encoded feature corresponding to the encoded image. Further, the first device inputs the third encoded feature into a mask for masking processing to obtain a first matrix.

[0088] In some alternative embodiments, the mask can mask the region corresponding to the coded sub-image in the third mask feature, or it can mask the region corresponding to one or more sub-images according to its own logic.

[0089] The first coding feature, the second coding feature, and the first matrix are input into the classification coding layer, and the output of the classification coding layer is confirmed to be the anti-enhancement coding sub-image corresponding to the coding sub-image. All coding sub-images are input into the class weight network to obtain the corresponding anti-enhancement coding sub-images. All anti-enhancement coding sub-images are concatenated to obtain an anti-enhancement coding image. After passing through the convolutional decoder, an anti-enhancement sample image is obtained.

[0090] Step S103: Adjust the parameters of the visual transformation network and the category weight network based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image.

[0091] In some embodiments, the first device determines the loss function of the image processing model based on the value of each pixel in the standard sample image and the value of each pixel in the anti-enhanced sample image; and adjusts the parameters of the visual transformation network and the category weight network based on the loss function of the image processing model. Specifically, the loss function of the image processing model may include a reconstruction loss sub-function, a perception loss sub-function, and a generator loss sub-function, and the loss function L of the image processing model can be expressed by the following formula:

[0092] L = L pixel +L per +L GAN

[0093] Among them, L pixel To reconstruct the loss function, L per For the perceptual loss subfunction, L GANThe generator loss function is used. Optionally, the loss function of the image processing model can also be determined based on the reconstruction loss function, the perception loss function, the generator loss function, and the weight coefficients corresponding to the three sub-functions, wherein the sum of the weight coefficients corresponding to the three sub-functions is 1.

[0094] In specific implementation, the first device determines the reconstruction loss function based on the value of each pixel in the standard sample image, the value of each pixel in the anti-enhanced sample image, and the L1 loss function. The reconstruction loss function can be expressed by the following formula:

[0095] L pixel =||I gt -I g ||1

[0096] Among them, I g For the value of each pixel in the anti-enhanced sample image, I gt This represents the value of each pixel in the standard sample image.

[0097] In specific implementation, the first device determines the perceptual loss function based on the value of each pixel in the standard sample image, the value of each pixel in the anti-enhanced sample image, and the visualized activation heatmap. The perceptual loss function can be expressed by the following formula:

[0098] L per =||Φ n (I gt )-Φ n (I g )||1

[0099] Where, Φ n This is the activation map of the nth layer of VGG (visualized activation heatmap).

[0100] In specific implementation, the first device determines the generator loss function based on the value of each pixel in the anti-enhanced sample image; the generator loss function can be expressed by the following formula:

[0101] L GAN =log(1+exp(-D(I)) g )))

[0102] Where D is the generator loss function of StyleGANv2.

[0103] Thus, through the training method of the image processing model provided in this embodiment, images collected under different conditions (such as lighting / dynamic obstacles / rain / fog / snow) can be uniformly formed into a standard image. On the one hand, this can reduce the time required for high-precision visual map production, because it is no longer necessary to collect image data under different environmental conditions for mapping; on the other hand, it can improve the robustness of real-time positioning, because environmental conditions will no longer cause differences between the positioning image and the map image.

[0104] Figure 5 A schematic diagram of another optional structure of the category weight network provided in this disclosure is shown, and will be described in terms of its various parts.

[0105] like Figure 5 As shown, the category weight network includes a first neural network layer, a second neural network layer, a third neural network layer, a mask, a classification encoding layer, and a convolutional decoder.

[0106] In some embodiments, taking a standard sample image size of 720×720 as an example, the convolution kernel size (size) of the first neural network layer, the second neural network layer, and the third neural network layer is 3×3, the stride is 0, and the activation function is Gelu.

[0107] In some embodiments, the first device inputs the encoded sub-image into a first neural network layer included in the class weight network to obtain a first encoded feature Xe corresponding to the encoded sub-image, with a size of B×C×D; and inputs the cropped sub-image corresponding to the encoded sub-image into a second neural network layer included in the class weight network to obtain a second encoded feature Xo corresponding to the cropped image, with a size of B×C×D. Wherein, C is the number of convolutional kernels, such as 256.

[0108] Then, the first device processes the first encoded feature Xe to obtain the autocorrelation strength A of the first encoded feature, wherein the autocorrelation strength A can be determined by the following formula:

[0109] A = Φ(Xe) T θ(Xe)

[0110] Where Φ and θ are both 1×1 convolution kernels. Where A has dimensions of B×D×D.

[0111] In some embodiments, the encoded image, after passing through a multilayer perceptron (MLP), is input to the third neural network layer to obtain a third encoded feature. The dimension of the third encoded feature is D×D / 8. The third encoded feature is then input to a mask and expanded to obtain a D×D dimensional matrix, i.e., the first matrix K. Where D = HW, H = 16, and W = 16.

[0112] In some embodiments, the first device divides the autocorrelation intensity into a non-masked region Av and a masked region Am based on a first matrix. Specifically,

[0113] Av=K·A

[0114] Am=(1-K)·A

[0115] Then, the first device processes the non-masked region Av and the masked region Am, which are divided by the autocorrelation intensity A based on Softmax. The purpose is to distinguish between dynamic obstacles and static obstacles. For dynamic obstacles, it is desirable to use as little of the original image as possible and generate as much as possible from images of other regions.

[0116] After processing the unmasked region Av with Softmax, the pixel weighting weight Softmax(Av) of the unmasked region is obtained. Combining Softmax(Av) with the second encoded feature Xo, the pixel weighting output Zv of the unmasked region is confirmed to be Zv = softmax(Av) × Xo. When Xo ij When dealing with static obstacles, we want the output of Softmax(Aij) to be as close to 1 as possible.

[0117] After processing the mask region Am with Softmax, the pixel weighting weight Softmax(Am) of the mask region is obtained. Combining Softmax(Am) and the first coding feature Xe, the pixel weighting output Ze of the mask region is confirmed to be softmax(Am)×Xe.

[0118] In some embodiments, the first device determines a first concatenation coefficient and a second concatenation coefficient based on the non-masked region and the masked region; specifically:

[0119] [Wv, Wm] = softmax [γ (max (Av), ζ (max (Am)))]

[0120] In this context, γ and ζ are both 1x1 convolution kernels, and Wv + Wm = 1. The dimensions of Wv and Wm are both B × 1 × H × W.

[0121] In some embodiments, the device determines the inverse enhancement coded sub-image Z corresponding to the cropped sub-image based on the first stitching coefficient Wv, the second stitching coefficient Wm, the pixel-weighted output Zv of the non-masked region, and the pixel-weighted output Zm of the masked region; wherein Z = Wv●Zv + Wm●Zm. Wv and Wm are extended to the C channel via broadcast. The dimensions of Zv and Zm are B×C×H×W, and the final generated Z has dimensions of B×C×H×W.

[0122] In some embodiments, the first device confirms the stitching result of the anti-enhancement coded sub-image corresponding to each cropped sub-image as the anti-enhancement coded sample image; and performs convolutional decoding on the anti-enhancement sample image to obtain the anti-enhancement sample image.

[0123] Figure 6 A schematic diagram illustrating the effect of image processing based on the image processing model provided in the embodiments of this disclosure is shown.

[0124] like Figure 6 As shown, the image processing model provided in this embodiment is used to remove occlusions from an image, adjust the brightness and clarity of the image to correspond to a high-precision visual feature map, and the extracted image feature points have a higher probability of successfully matching with image feature points in the high-precision visual feature map to obtain the real-time pose of the current vehicle or robot.

[0125] Figure 7 This illustration shows an optional flowchart of the positioning method provided in an embodiment of the present disclosure. Figure 8 A schematic diagram of the localization method provided in an embodiment of this disclosure is shown. The localization method is implemented based on the image processing model trained in steps S101 to S103 described above.

[0126] Step S201: Based on the image processing model, normalize the real-time image and the constructed image respectively to obtain the real-time features corresponding to the real-time image and the constructed features corresponding to the constructed image respectively.

[0127] In some embodiments, the positioning device (hereinafter referred to as the second device) performs normalization processing on the real-time image and the mapping image respectively to obtain an implementation enhancement image and a mapping enhancement image with the same occlusion situation, brightness and sharpness; and obtains the real-time features corresponding to the implementation enhancement image (i.e., the real-time features corresponding to the real-time image) and the mapping features corresponding to the mapping enhancement image (i.e., the mapping features corresponding to the mapping image).

[0128] Step S202: Perform feature fusion on the global navigation satellite system and mapping features corresponding to the real-time image, and confirm the feature map based on the feature fusion result.

[0129] The GNSS corresponding to the second device is fused with the mapping features, and a feature map (i.e., a high-precision visual feature map) is obtained based on the fusion result.

[0130] Step S203: Based on the feature map and the real-time features, confirm the location information of the device that acquired the real-time image.

[0131] In some embodiments, the second device matches the feature points corresponding to the real-time features with the feature points included in the feature map, and based on the matching results, confirms the location information of the device that acquired the real-time image.

[0132] Thus, by adjusting the brightness and clarity of the real-time image using the image processing model provided in this embodiment, the image corresponds to the map image of the high-precision visual feature map; the extracted real-time features have a higher probability of successfully matching the image features in the high-precision visual feature map, thereby obtaining the real-time pose of the current vehicle or robot.

[0133] Figure 9 A schematic diagram of an optional structure of a training apparatus for an image processing model provided in an embodiment of this disclosure is shown.

[0134] In some embodiments, the image processing model includes a visual transformation network and a category weight network; the training device 400 of the image processing model includes a first training unit 401, a second training unit 402, and an adjustment unit 403.

[0135] The first training unit 401 is used to input at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image respectively;

[0136] The second training unit 402 is used to input any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by stitching together all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image;

[0137] The adjustment unit 403 is used to adjust the parameters of the visual transformation network and the category weight network based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image.

[0138] In some embodiments, the training apparatus 400 for the image processing model may further include a processing unit 404.

[0139] Before inputting at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the coded sub-images corresponding to the at least one cropped sub-image, the processing unit 404 performs preprocessing on the original sample image to obtain a standard sample image; enhances the standard sample image to obtain an enhanced sample image; and crops the enhanced sample image to obtain at least one cropped sub-image corresponding to the enhanced sample image.

[0140] The first training unit 401 is specifically used to input the at least one cropped sub-image into the linear layer included in the visual transformation network, and to confirm that the output of the linear layer is the linear projection corresponding to each cropped sub-image.

[0141] The linear projection corresponding to each cropped sub-image and the classification identifier of each cropped sub-image in the enhanced sample image are input into the encoder included in the visual transformation network, and the output of the encoder is confirmed to be the coded sub-image corresponding to each of the at least one cropped sub-image.

[0142] The second training unit 402 is specifically used to perform the following processing on each coded sub-image and the corresponding cropped sub-image:

[0143] The encoded sub-image is input into the first neural network layer of the category weight network to obtain the first encoded feature corresponding to the encoded sub-image;

[0144] The cropped sub-image corresponding to the encoded sub-image is input into the second neural network layer of the category weight network to obtain the second encoded feature corresponding to the cropped image;

[0145] The encoded image is input into the third neural network layer of the category weight network to obtain the third encoded feature corresponding to the encoded image.

[0146] The second training unit 402 is specifically used to perform the following processing on each coded sub-image and the corresponding cropped sub-image:

[0147] Based on the first coding feature, the autocorrelation strength of the first coding feature is obtained;

[0148] The third encoded feature is masked to obtain the first matrix;

[0149] The autocorrelation intensity is divided into a non-masked region and a masked region based on the first matrix.

[0150] The second training unit 402 is specifically used to perform the following processing on each coded sub-image and the corresponding cropped sub-image:

[0151] Based on the pixel weighting weights of the non-masked region and the second encoding feature, the pixel weighting output of the non-masked region is confirmed;

[0152] Based on the pixel weighting of the masked region and the first encoding feature, the pixel weighting output of the masked region is confirmed;

[0153] Based on the unmasked region and the masked region, the first and second combining coefficients are determined;

[0154] Based on the first concatenation coefficient, the second concatenation coefficient, the pixel weighted output of the non-masked region, and the pixel weighted output of the masked region, the inverse enhancement coded sub-image corresponding to the cropped sub-image is confirmed;

[0155] The stitching result of the inverse-enhanced coded sub-image corresponding to each cropped sub-image is confirmed as the inverse-enhanced coded sample image;

[0156] The anti-enhanced sample image is obtained by performing convolutional decoding on the anti-enhanced sample image.

[0157] The adjustment unit 403 is specifically used to determine the loss function of the image processing model based on the value of each pixel in the standard sample image and the value of each pixel in the anti-enhanced sample image;

[0158] Based on the loss function of the image processing model, the parameters of the visual transformation network and the category weight network are adjusted.

[0159] The adjustment unit 403 is specifically used to determine the reconstruction loss sub-function based on the value of each pixel in the standard sample image, the value of each pixel in the anti-enhanced sample image, and the L1 loss function;

[0160] Based on the value of each pixel in the standard sample image, the value of each pixel in the anti-enhanced sample image, and the visualized activation heatmap, the perceptual loss sub-function is confirmed;

[0161] Based on the value of each pixel in the anti-enhanced sample image, the generator loss function is determined;

[0162] Based on the reconstruction loss function, the perception loss function, and the generator loss function, the loss function of the image processing model is determined.

[0163] Figure 10 A schematic diagram of an optional structure of the positioning device provided in an embodiment of this disclosure is shown, and the details will be described in terms of each part.

[0164] In some embodiments, the positioning device 500 includes a normalization unit 501, a feature fusion unit 502, and a positioning unit 503.

[0165] The normalization unit 501 is used to perform normalization processing on the real-time image and the constructed image based on the image processing model, respectively, to obtain the real-time features corresponding to the real-time image and the constructed features corresponding to the constructed image.

[0166] The feature fusion unit 502 is used to perform feature fusion on the Global Navigation Satellite System (GNSS) and mapping features corresponding to the real-time image, and to confirm the feature map based on the feature fusion result.

[0167] The positioning unit 503 is used to confirm the positioning information of the device that acquired the real-time image based on the feature map and the real-time features.

[0168] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0169] Figure 11 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0170] like Figure 11 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0171] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0172] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as image processing model training and localization methods. For example, in some embodiments, the image processing model training and localization methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the image processing model training and localization methods described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform training and localization methods for the image processing model.

[0173] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0175] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0177] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0178] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0179] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0180] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0181] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A training method for an image processing model, characterized in that, The image processing model includes a visual transformation network and a category weight network, and the method includes: At least one cropped sub-image corresponding to the enhanced sample image is input into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image; Any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by stitching together all the coded sub-images are input into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image; the category weight network includes a first neural network layer, a third neural network layer, a mask, a classification coding layer, and a convolutional decoder. The parameters of the visual transformation network and the category weight network are adjusted based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image.

2. The method according to claim 1, characterized in that, Before inputting at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image, the method further includes: Preprocess the original sample images to obtain standard sample images; The standard sample image is enhanced to obtain an enhanced sample image; The enhanced sample image is cropped to obtain at least one cropped sub-image corresponding to the enhanced sample image.

3. The method according to claim 1, characterized in that, The step of inputting at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image includes: The at least one cropped sub-image is input into the linear layer of the visual transformation network, and the output of the linear layer is confirmed to be the linear projection corresponding to each cropped sub-image. The linear projection corresponding to each cropped sub-image and the classification identifier of each cropped sub-image in the enhanced sample image are input into the encoder included in the visual transformation network, and the output of the encoder is confirmed to be the coded sub-image corresponding to each of the at least one cropped sub-image.

4. The method according to claim 1, characterized in that, The step of inputting any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by concatenating all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image includes performing the following processing on each coded sub-image and the cropped sub-image corresponding to each coded sub-image: The encoded sub-image is input into the first neural network layer of the category weight network to obtain the first encoded feature corresponding to the encoded sub-image; The cropped sub-image corresponding to the encoded sub-image is input into the second neural network layer of the category weight network to obtain the second encoded feature corresponding to the cropped image; The encoded image is input into the third neural network layer of the category weight network to obtain the third encoded feature corresponding to the encoded image.

5. The method according to claim 4, characterized in that, The step of inputting any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by concatenating all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image includes performing the following processing on each coded sub-image and the cropped sub-image corresponding to each coded sub-image: Based on the first coding feature, the autocorrelation strength of the first coding feature is obtained; The third encoded feature is masked to obtain the first matrix; The autocorrelation intensity is divided into a non-masked region and a masked region based on the first matrix.

6. The method according to claim 5, characterized in that, The step of inputting any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by concatenating all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image includes performing the following processing on each coded sub-image and the cropped sub-image corresponding to each coded sub-image: Based on the pixel weighting weights of the non-masked region and the second encoding feature, the pixel weighting output of the non-masked region is confirmed; Based on the pixel weighting of the masked region and the first encoding feature, the pixel weighting output of the masked region is confirmed; Based on the unmasked region and the masked region, the first and second combining coefficients are determined; Based on the first concatenation coefficient, the second concatenation coefficient, the pixel weighted output of the non-masked region, and the pixel weighted output of the masked region, the inverse enhancement coded sub-image corresponding to the cropped sub-image is confirmed; The stitching result of the inverse-enhanced coded sub-image corresponding to each cropped sub-image is confirmed as the inverse-enhanced coded sample image; The anti-enhanced sample image is obtained by performing convolutional decoding on the anti-enhanced sample image.

7. The method according to claim 1, characterized in that, The step of adjusting the parameters of the visual transformation network and the category weight network based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image includes: Based on the value of each pixel in the standard sample image and the value of each pixel in the anti-enhanced sample image, the loss function of the image processing model is determined; Based on the loss function of the image processing model, the parameters of the visual transformation network and the category weight network are adjusted.

8. The method according to claim 7, characterized in that, The step of determining the loss function of the image processing model based on the value of each pixel in the standard sample image and the value of each pixel in the anti-enhanced sample image includes: Based on the value of each pixel in the standard sample image, the value of each pixel in the anti-enhanced sample image, and the L1 loss function, the reconstruction loss sub-function is determined; Based on the value of each pixel in the standard sample image, the value of each pixel in the anti-enhanced sample image, and the visualized activation heatmap, the perceptual loss sub-function is confirmed; Based on the value of each pixel in the anti-enhanced sample image, the generator loss function is determined; Based on the reconstruction loss function, the perception loss function, and the generator loss function, the loss function of the image processing model is determined.

9. A positioning method, characterized in that, The method, implemented based on the image processing model trained according to claims 1-8, includes: Based on the image processing model, the real-time image and the constructed image are normalized respectively to obtain the real-time features corresponding to the real-time image and the constructed features corresponding to the constructed image. The GNSS and mapping features corresponding to the real-time image are fused, and the feature map is confirmed based on the result of the feature fusion. Based on the feature map and the real-time features, the location information of the device that acquired the real-time image is confirmed.

10. A training device for an image processing model, characterized in that, The image processing model includes a visual transformation network and a category weight network, and the device includes: The first training unit is used to input at least one cropped sub-image corresponding to the enhanced sample image into the visual transformation network to obtain the encoded sub-images corresponding to the at least one cropped sub-image respectively; The second training unit is used to input any coded sub-image, the cropped sub-image corresponding to the coded sub-image, and the coded image obtained by stitching together all the coded sub-images into the category weight network to obtain the anti-enhancement sample image corresponding to the enhanced sample image; the category weight network includes a first neural network layer, a third neural network layer, a mask, a classification coding layer, and a convolutional decoder. The adjustment unit is used to adjust the parameters of the visual transformation network and the category weight network based on the standard sample image corresponding to the enhanced sample image and the anti-enhanced sample image.

11. A positioning device, characterized in that, The device, based on the image processing model trained according to claims 1-8, comprises: The normalization unit is used to perform normalization processing on the real-time image and the constructed image based on the image processing model, respectively, to obtain the real-time features corresponding to the real-time image and the constructed features corresponding to the constructed image; The feature fusion unit is used to perform feature fusion between the Global Navigation Satellite System (GNSS) and mapping features corresponding to the real-time image, and to confirm the feature map based on the feature fusion result. A positioning unit is used to confirm the positioning information of the device that acquired the real-time image based on the feature map and the real-time features.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the method of any one of claims 1-8; Alternatively, the method described in claim 9 can be performed.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to enable the computer to perform the method according to any one of claims 1-8; Alternatively, the method described in claim 9 can be performed.

Images