Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first initial image and a second initial image; preprocessing the first initial image and the second initial image to obtain a first image after the first initial image is processed and a second image after the second initial image is processed; the visual angle difference between the first image and the second image is smaller than a first threshold, and the scale difference is smaller than a second threshold; extracting a first feature map of the first image and a second feature map of the second image, and extracting a reference feature point pair based on the first feature map and the second feature map; and based on the reference feature point pair, correcting the first image and the second image to obtain a first target image after the first image is corrected and a second target image after the second image is corrected. According to the method, the difficulty of extracting the image features and the reference feature point pairs is reduced, and more accurate reference feature point pairs can be extracted, so that the image correction precision is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] In electronic devices that use multiple camera modules to capture images, traditional image correction algorithms struggle to accurately correct the left and right images when shooting complex scenes with weak textures, repetitive textures, or high noise. This directly impacts the performance of the corrected images in subsequent applications such as depth perception and 3D reconstruction. This is especially true in dual-camera systems with small baselines, where even minor differences between the left and right images exacerbate the difficulty of image correction. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising:

[0005] Acquire a first initial image and a second initial image; the first initial image and the second initial image are captured by two camera modules in the same electronic device on the same shooting scene;

[0006] The first initial image and the second initial image are preprocessed to obtain a first image after processing the first initial image and a second image after processing the second initial image; the difference in viewpoint between the first image and the second image is less than a first threshold and the difference in scale is less than a second threshold.

[0007] Extract a first feature map from the first image and a second feature map from the second image, and extract reference feature point pairs based on the first feature map and the second feature map; the reference feature point pairs include a first reference feature point in the first feature map and a second reference feature point in the second feature map that matches the first reference feature point;

[0008] Based on the reference feature point pair, the first image and the second image are corrected to obtain a first target image after the first image is corrected and a second target image after the second image is corrected.

[0009] In some embodiments, the preprocessing of the first initial image and the second initial image to obtain a first image after processing the first initial image and a second image after processing the second initial image includes:

[0010] The first initial image and the second initial image are cropped to obtain the first image and the second image.

[0011] In some embodiments, before extracting the first feature map of the first image and the second feature map of the second image, the method further includes:

[0012] Image quality correction is performed on the first image and the second image to ensure that the image quality difference between the corrected first image and the corrected second image is less than a third threshold; and / or,

[0013] The first image and the second image are subjected to distortion correction processing respectively to obtain the first image after distortion correction and the second image after distortion correction.

[0014] In some embodiments, extracting a first feature map of the first image and a second feature map of the second image, and extracting reference feature point pairs based on the first feature map and the second feature map, includes:

[0015] The target feature matching model is invoked to extract the first feature map of the first image and the second feature map of the second image, and the reference feature point pair is extracted based on the first feature map and the second feature map; the target feature matching model is used to extract the reference feature point pair corresponding to the two input images.

[0016] In some embodiments, the target feature matching model includes a feature extraction network and a feature matching layer; the step of calling the target feature matching model to extract a first feature map of the first image and a second feature map of the second image, and extracting the reference feature point pair based on the first feature map and the second feature map, includes:

[0017] The feature extraction network is invoked to extract features from the first image and the second image respectively, resulting in the first feature map and the second feature map.

[0018] The feature matching layer is invoked to extract the reference feature point pair based on the first feature map and the second feature map.

[0019] In some embodiments, the feature extraction network includes a first encoder, a second encoder, and a fusion layer; the step of invoking the feature extraction network to extract features from the first image and the second image respectively to obtain the first feature map and the second feature map includes:

[0020] The first encoder is invoked to extract features from the first image to obtain a third feature map, and features are extracted from the second image to obtain a fourth feature map; the size of the third feature map and the fourth feature map are both a first size, and the number of channels contained in both is a first number;

[0021] The second encoder is invoked to extract features from the third feature map to obtain the fifth feature map, and to extract features from the fourth feature map to obtain the sixth feature map; the fifth feature map and the sixth feature map are both the first size, and both contain the second number of channels, wherein the second number is greater than the first number;

[0022] The fusion layer is invoked to fuse the third feature map and the fifth feature map to obtain the first feature map, and to fuse the fourth feature map and the sixth feature map to obtain the second feature map; the size of the first feature map and the second feature map are both the first size, and the number of channels contained are both the second number.

[0023] In some embodiments, invoking the feature matching layer to extract reference feature point pairs based on the first feature map and the second feature map includes:

[0024] The feature matching layer is invoked to perform the following steps:

[0025] The first feature map is divided into multiple first feature blocks, and the second feature map is divided into multiple second feature blocks. Based on the multiple first feature blocks and the multiple second feature blocks, multiple feature block pairs are determined. Each feature block pair includes a first feature block and a second feature block corresponding to the first feature block. Each first feature block includes N first feature points, and each second feature block includes N second feature points, where N is an integer greater than 1.

[0026] For each feature block pair, the feature distance between each first feature point in the first feature block included in the feature block pair and each second feature point in the second feature block included in the feature block pair is determined. Based on the determined N×N feature distances, a reference feature point pair corresponding to the feature block pair is determined. The feature distance is used to characterize the similarity between the first feature point and the second feature point.

[0027] In some embodiments, determining the reference feature point pair corresponding to the feature block pair based on the determined N×N feature distances includes:

[0028] Based on the N×N feature distances, a confidence matrix is ​​determined; the confidence matrix is ​​an N-row N-column matrix, and the element in the i-th row and j-th column of the confidence matrix is ​​used to characterize the feature distance between the i-th first feature point among the N first feature points and the j-th second feature point among the N second feature points, where i and j are integers less than or equal to N;

[0029] Determine the centroid location in the confidence matrix;

[0030] The centroid position is converted into the coordinates of the first reference feature point and the second reference feature point in the reference feature point pair.

[0031] In some embodiments, determining the confidence matrix based on the N×N feature distances includes:

[0032] Based on the N×N feature distances, a mutual distance matrix is ​​determined; the mutual distance matrix is ​​an N-row N-column matrix, and the element in the i-th row and j-th column of the mutual distance matrix is ​​the feature distance between the i-th first feature point among the N first feature points and the j-th second feature point among the N second feature points;

[0033] The confidence matrix is ​​obtained by normalizing each element in the mutual distance matrix.

[0034] In some embodiments, the training process of the target feature matching model includes:

[0035] Obtain a training dataset; the training dataset includes multiple first sample images, a second sample image corresponding to each first sample image, and multiple feature point pair labels, wherein the feature point pair labels include first sample feature points in the first sample images and second sample feature points in the second sample images corresponding to the first sample images that correspond to the first sample images;

[0036] Based on the training dataset, a feature matching model to be trained is trained to obtain the target feature matching model.

[0037] In some embodiments, obtaining the training dataset includes:

[0038] A first depth image and a second depth image are acquired, wherein the first depth image and the second depth image are captured by two camera modules in the same depth camera on the same shooting scene;

[0039] Determine the correspondence between pixels in the first depth image and pixels in the second depth image;

[0040] The first depth image and the second depth image are cropped to obtain the first sample image and the second sample image; the baseline length between the first sample image and the second sample image conforms to the baseline length of the two camera modules in the electronic device.

[0041] Based on the correspondence, the first sample feature point in the first sample image and the second sample feature point in the second sample image corresponding to the first sample feature point are determined.

[0042] In some embodiments, the step of correcting the first image and the second image based on the reference feature point pair to obtain a first target image after correction of the first image and a second target image after correction of the second image includes:

[0043] The reference feature point pair is mapped to the first image and the second image to obtain the target feature point pair; the target feature point pair includes a first target feature point in the first image and a second target feature point in the second image that matches the first target feature point.

[0044] Based on the target feature point pair, the first image and the second image are corrected to obtain the first target image and the second target image.

[0045] In some embodiments, the step of correcting the first image and the second image based on the target feature point pair to obtain the first target image and the second target image includes:

[0046] Using a preset algorithm, based on the target feature point pairs and the initial intrinsic and extrinsic parameters of each camera module, the target intrinsic and extrinsic parameters of each camera module are determined;

[0047] Based on the target intrinsic and target extrinsic parameters of each camera module, the first image and the second image are row aligned to obtain the first target image and the second target image.

[0048] According to a second aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0049] The initial image acquisition module is configured to acquire a first initial image and a second initial image, wherein the first initial image and the second initial image are captured by two camera modules in the same electronic device on the same shooting scene;

[0050] The image preprocessing module is configured to preprocess the first initial image and the second initial image to obtain a first image after processing the first initial image and a second image after processing the second initial image; the viewpoint difference between the first image and the second image is less than a first threshold and the scale difference is less than a second threshold.

[0051] The feature matching module is configured to extract a first feature map of the first image and a second feature map of the second image, and extract reference feature point pairs based on the first feature map and the second feature map; the reference feature point pairs include a first reference feature point in the first feature map and a second reference feature point in the second feature map that matches the first reference feature point;

[0052] The image correction module is configured to correct the first image and the second image based on the reference feature point pair to obtain a first target image after correction of the first image and a second target image after correction of the second image.

[0053] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0054] processor;

[0055] Memory used to store processor-executable instructions;

[0056] The processor is configured to perform the image processing method as described in the first aspect of the embodiments of this disclosure.

[0057] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described in the first aspect of the present disclosure.

[0058] The method described above, as disclosed in this invention, has the following beneficial effects:

[0059] The method provided in this embodiment addresses the significant differences in perspective and scale between the first and second initial images directly captured by the electronic device. Before image correction, the first and second initial images are preprocessed to reduce these differences, resulting in more similar images. This reduces the difficulty of feature extraction and reference feature point pair extraction, leading to improved image correction accuracy.

[0060] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0062] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0063] Figure 2 This is a schematic diagram illustrating an input image and an effect image according to an exemplary embodiment;

[0064] Figure 3 This is a flowchart illustrating a feature extraction process according to an exemplary embodiment;

[0065] Figure 4 This is a schematic diagram illustrating a network structure according to an exemplary embodiment;

[0066] Figure 5 This is a schematic diagram illustrating a feature matching effect according to an exemplary embodiment;

[0067] Figure 6 This is a schematic diagram illustrating a feature matching process according to an exemplary embodiment;

[0068] Figure 7 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0069] Figure 8 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment;

[0070] Figure 9 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0072] Most mainstream solutions in the field of image correction employ feature matching algorithms based on traditional computer vision, such as ORB (Oriented Fast and Rotated BRIEF) and SURF (Speeded Up Robust Features). These algorithms extract and match local feature points in an image to generate the correspondence between the left and right images, and then complete image correction through geometric transformation. Feature matching algorithms based on traditional computer vision perform well in terms of computational complexity and real-time performance, making them suitable for resource-constrained environments. However, in complex scenes with weak textures, repetitive textures, and high noise, the accuracy of feature point extraction and matching is limited, making it difficult to generate sufficiently accurate correction results.

[0073] This disclosure provides an image processing method that involves acquiring a first initial image and a second initial image; preprocessing the first and second initial images to obtain a first image after processing the first initial image and a second image after processing the second initial image; ensuring that the viewpoint difference between the first and second images is less than a first threshold and the scale difference between the first and second images is less than a second threshold; extracting a first feature map from the first image and a second feature map from the second image, and extracting reference feature point pairs based on the first and second feature maps; and correcting the first and second images based on the reference feature point pairs to obtain a first target image after correcting the first image and a second target image after correcting the second image. Before correcting the images, preprocessing the first and second initial images reduces the viewpoint and scale differences between the preprocessed first and second images, resulting in more similar first and second images. This reduces the difficulty of feature extraction and reference feature point pair extraction on the first and second images, allowing for the extraction of more accurate reference feature point pairs, thereby improving the accuracy of image correction.

[0074] The image processing method provided in this disclosure is executed by an electronic device, which may be a mobile phone, tablet computer, laptop computer, wearable device, smart home device, drone, smart glasses, vehicle terminal, or other such device.

[0075] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment, executed by an electronic device, see [link to flowchart]. Figure 1 The method includes the following steps:

[0076] Step S101: Obtain a first initial image and a second initial image. The first initial image and the second initial image are obtained by two camera modules in the same electronic device capturing the same shooting scene.

[0077] The electronic device includes at least two camera modules. The first initial image and the second initial image are captured by any two of the camera modules of the at least two camera modules on the same shooting scene. These two camera modules can capture the first initial image and the second initial image simultaneously, or they can capture the first initial image first and then the second initial image, or vice versa. Furthermore, this embodiment does not limit the shooting scene; the first initial image and the second initial image can be captured from any shooting scene.

[0078] Step S102: Preprocess the first initial image and the second initial image to obtain the first image after processing the first initial image and the second image after processing the second initial image; the difference in viewpoint between the first image and the second image is less than the first threshold and the difference in scale is less than the second threshold.

[0079] Because the two camera modules are located in different positions within the electronic device, the field of view (FOV) and scale of the first and second initial images will differ. However, since the first and second initial images were captured from the same scene, they will also contain identical image content. Therefore, based on these characteristics, the first and second initial images are preprocessed to ensure that the FOV difference between the processed images is less than a first threshold and the scale difference is less than a second threshold. Here, the FOV difference refers to the difference between the FOV of the first image and the FOV of the second image; the scale difference refers to the difference between the dimensions of the first image and the second image, as well as the difference between the dimensions of a target object in the first image and the second image. The target object refers to any object contained in both the first and second images. The first and second thresholds are pre-set, relatively small thresholds. The smaller these thresholds are, the more similar the first and second images will be, and the simpler the subsequent processing will be.

[0080] Step S103: Extract the first feature map of the first image and the second feature map of the second image, and extract reference feature point pairs based on the first feature map and the second feature map; the reference feature point pairs include the first reference feature point in the first feature map and the second reference feature point in the second feature map that matches the first reference feature point.

[0081] The first feature map characterizes the image features of the first image, and the second feature map characterizes the image features of the second image. These image features can refer to features describing the image, such as contour features, semantic features, and texture features. The first and second feature maps have the same size and the same number of channels. Reference feature point pairs characterize the matching relationship between feature points in the first image and feature points in the second image.

[0082] Since the first and second images are already quite similar, when extracting the first and second feature maps, there is no need to use complex feature extraction methods to extract accurate image features. Based on this, when extracting reference feature point pairs based on the first and second feature maps, even more accurate reference feature point pairs can also be extracted.

[0083] Step S104: Based on the reference feature point pair, the first image and the second image are corrected to obtain the first target image after the first image is corrected and the second target image after the second image is corrected.

[0084] Since the reference feature point pair has already characterized the matching relationship between the feature points in the first image and the feature points in the second image, the first image and the second image can be corrected based on the reference feature point pair so that the corrected first target image and the second target image are aligned in the row direction.

[0085] It should be noted that there can be multiple extracted reference feature point pairs. Then, based on the multiple reference feature point pairs, the first image and the second image are corrected to obtain the first target image and the second target image.

[0086] The method provided in this embodiment addresses the significant differences in perspective and scale between the first and second initial images directly captured by the electronic device. Before image correction, the first and second initial images are preprocessed to reduce these differences, resulting in more similar images. This reduces the difficulty of feature extraction and reference feature point pair extraction, leading to improved image correction accuracy.

[0087] In some embodiments, the preprocessing of the first initial image and the second initial image includes cropping the first initial image and the second initial image to obtain a first image and a second image. During the cropping process, the shooting parameters of the two camera modules can be referenced, such as the focal length, viewing angle, and baseline length during shooting, to ensure that the viewing angle difference between the cropped first image and the second image is less than a first threshold and the scale difference is less than a second threshold.

[0088] In some embodiments, due to hardware differences between the two camera modules and the influence of ambient light, the image quality of the first initial image and the second initial image may differ significantly. In this case, after cropping, image quality correction is performed on both the first and second images to ensure that the image quality difference between the corrected first and second images is less than a third threshold. The third threshold is a pre-set, relatively small threshold; the smaller the third threshold, the smaller the image quality difference between the corrected first and second images. Optionally, algorithms such as bilateral noise reduction or joint histogram equalization of the left and right images can be used for image quality correction.

[0089] In some embodiments, during actual shooting, the lens in the camera module may introduce geometric distortions, such as barrel distortion or pincushion distortion. These distortions cause features that should be straight lines to appear curved in the image, thus affecting the accuracy of subsequent feature point extraction and matching. Furthermore, distortion can lead to non-rigid transformations in the image, increasing the image row alignment error. Distortion removal processing can eliminate these distortions, thereby improving the accuracy of subsequent image correction. Therefore, distortion removal processing is performed on the first image and the second image respectively to obtain the distorted first image and the distorted second image.

[0090] Optionally, a distortion model is determined based on the lens distortion type; distortion coefficients are obtained using a camera calibration method; and distortion correction processing is performed on the first and second images based on the distortion model and distortion coefficients. The camera calibration method can be Zhang's calibration method or other calibration methods, and this embodiment does not limit the specific methods used.

[0091] In one example, the following formula is used to represent the correction for radial distortion at any pixel in the image:

[0092] r corrected = r×(1+k1r2+k2r) 4 )

[0093] Where, r corrected The distance from the corrected pixel to the preset center is represented by , where r represents the distance from the uncorrected pixel to the preset center, and k1 and k2 are distortion coefficients. The preset center can be the center point of the image.

[0094] It should be noted that the embodiments disclosed herein are only illustrated by taking the example of first cropping the first initial image and the second initial image, and then performing image quality correction and distortion correction. In another embodiment, image quality correction and / or distortion correction may be performed on the first initial image and the second initial image first, and then cropping may be performed. The embodiments disclosed herein do not limit the order of cropping, image quality correction and distortion correction.

[0095] In some embodiments, a first image and a second image are corrected based on a reference feature point pair to obtain a first target image after the first image is corrected and a second target image after the second image is corrected. This includes: mapping the reference feature point pair to the first image and the second image to obtain a target feature point pair; the target feature point pair includes a first target feature point in the first image and a second target feature point in the second image that matches the first target feature point; and correcting the first image and the second image based on the target feature point pair to obtain the first target image and the second target image.

[0096] Optionally, mapping the reference feature point pair to the first image and the second image to obtain the target feature point pair includes: determining the multiple by which the size of the first feature map differs from the size of the first image, and the multiple by which the size of the second feature map differs from the size of the second image; based on these multiples, adjusting the coordinates of the first reference feature point and the second reference feature point to obtain the coordinates of the first target feature point and the second target feature point, thereby determining the target feature point pair. For example, if the size of the first feature map and the second feature map is H / 8 × W / 8, and the size of the first image and the second image is H × W, then determining that the size of the first image and the second image is 8 times the size of the first feature map and the second feature map, then multiplying the coordinates of the first reference feature point and the second reference feature point by 8, yields the coordinates of the first target feature point and the second target feature point. Of course, other mapping methods can also be used, and this disclosure does not limit this approach.

[0097] Optionally, based on target feature point pairs, the first image and the second image are corrected to obtain a first target image and a second target image. This includes: using a preset algorithm, determining the target intrinsic and extrinsic parameters of each camera module based on the target feature point pairs and the initial intrinsic and extrinsic parameters of each camera module; and performing row alignment processing on the first image and the second image based on the target intrinsic and extrinsic parameters of each camera module to obtain the first target image and the second target image. The initial intrinsic and extrinsic parameters are determined based on the hardware design of the camera module, and the preset algorithm can be the Gauss-Newton method.

[0098] Optionally, the Gauss-Newton method can be used to iteratively solve for the initial intrinsic and extrinsic parameters of each camera module based on the target feature point pairs until satisfactory target intrinsic and extrinsic parameters are obtained. Of course, other methods can also be used to determine the target intrinsic and extrinsic parameters.

[0099] Optionally, after determining the target intrinsic and extrinsic parameters of the two camera modules—that is, determining the rotation and translation relationships between the two camera modules and the intrinsic parameter matrix of each camera module—a geometric transformation matrix is ​​determined. This geometric transformation matrix is ​​then used to reproject the first and second images to obtain the first and second target images, ensuring row alignment between the first and second target images. See one example. Figure 2 The diagram shows the input image and the result image. Figure 2 'a' represents the first and second input images. Figure 2 b represents the corrected first and second target images, from... Figure 2 As can be seen, the corrected image has excellent row alignment.

[0100] In related technologies, deep learning algorithms require strong feature representation and global perception capabilities when dealing with challenging problems such as large-viewpoint transformations, large-scale changes, and illumination differences between images. Therefore, feature encoding networks use deep layers for image feature extraction during network design. Simultaneously, to enable global perception, attention mechanisms are introduced, particularly multi-layered self-attention and cross-attention designs, to enhance the aggregation of global features. For example, the Loftr (Local Feature Transformers) algorithm achieves more comprehensive feature representation by adding a global attention mechanism, thereby improving the robustness and accuracy of feature matching. However, this design also leads to a more complex network structure, increasing computational overhead and placing higher demands on performance and power consumption. This presents significant challenges, especially for mobile devices with limited computing resources, where applying such complex networks presents considerable difficulties.

[0101] In this embodiment, a target feature matching model is invoked to extract a first feature map of the first image and a second feature map of the second image. Based on the first and second feature maps, reference feature point pairs are extracted. The target feature matching model is used to extract reference feature point pairs corresponding to the two input images. Since the first and second initial images are preprocessed, their viewpoints and scales are closer. Therefore, only a smaller network receptive field is needed for accurate feature extraction and matching. That is, the target feature matching model used in this embodiment has a simpler network result than the models in related technologies, can use fewer layers in the feature encoding network, and does not require the introduction of an attention mechanism.

[0102] In some embodiments, the target feature matching model includes a feature extraction network and a feature matching layer; calling the target feature matching model to extract a first feature map of the first image and a second feature map of the second image, and extracting reference feature point pairs based on the first feature map and the second feature map, includes: calling the feature extraction network to extract features from the first image and the second image respectively to obtain the first feature map and the second feature map; and calling the feature matching layer to extract reference feature point pairs based on the first feature map and the second feature map.

[0103] In some embodiments, the feature extraction network includes a first encoder, a second encoder, and a fusion layer. The process of invoking the feature extraction network to extract the first and second feature maps is as follows. Figure 3 As shown.

[0104] Figure 3 This is a flowchart illustrating a feature extraction process according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 3 This includes the following steps:

[0105] Step S301: Call the first encoder to extract features from the first image to obtain the third feature map, and extract features from the second image to obtain the fourth feature map.

[0106] In this design, both the third and fourth feature maps have a first size and a first number of channels. For example, if the first and second images have a size of H×W, then the extracted third and fourth feature maps can have a size of H / 8×W / 8 and 64 channels, meaning they are H / 8×W / 8×64 feature maps. The third feature map is used to characterize the contour, texture, and other features of the first image; similarly, the fourth feature map is used to characterize the contour, texture, and other features of the second image.

[0107] Since the first and second images are quite similar, the first encoder does not need a complex structure and contains fewer layers, thus reducing the required computational resources. In one example, the first encoder includes four blocks, each containing: a convk×k (k×k convolutional layer), a BN (normalization layer), and a ReLU (activation layer).

[0108] Step S302: Call the second encoder to extract features from the third feature map to obtain the fifth feature map, and extract features from the fourth feature map to obtain the sixth feature map.

[0109] In this design, both the fifth and sixth feature maps have a first size and contain a second number of channels, which is greater than the first number. For example, the first number is 64, and the second number can be 128. The second encoder performs further feature extraction on the feature maps extracted by the first encoder to obtain deeper features. For example, the fifth feature map is used to represent the semantics of the first image, and the sixth feature map is used to represent the semantics of the second image.

[0110] Similar to the first encoder, the second encoder does not require a overly complex structure. It includes fewer layers, reducing the computational resources required. In one example, the second encoder includes six blocks, each containing: a convk×k (k×k convolutional layer), a BN (normalization layer), and a ReLU (activation layer).

[0111] Step S303: Call the fusion layer to fuse the third and fifth feature maps to obtain the first feature map, and fuse the fourth and sixth feature maps to obtain the second feature map.

[0112] The first feature map and the second feature map are both of the first size, and both contain the second number of channels.

[0113] Optionally, when fusing the third and fifth feature maps, the number of channels in the third feature map can be increased to a second number, and then the feature points at corresponding positions in the third and fifth feature maps can be added to obtain the first feature map. Similarly, when fusing the fourth and sixth feature maps, the number of channels in the fourth feature map can be increased to a second number, and then the feature points at corresponding positions in the fourth and sixth feature maps can be added to obtain the second feature map. Of course, other fusion methods can also be used, and this embodiment does not limit the fusion method.

[0114] In this embodiment, two encoders are used for feature extraction, which can fully extract features from the first and second images, resulting in more accurate first and second feature maps. Furthermore, to improve the feature representation capability of the feature extraction network, a reparameterized network structure is used to achieve a balance between network accuracy and performance.

[0115] In some embodiments, the process of invoking the feature matching layer to extract reference feature point pairs includes: invoking the feature matching layer to perform the following steps: dividing the first feature map into multiple first feature blocks, dividing the second feature map into multiple second feature blocks, and determining multiple feature block pairs based on the multiple first feature blocks and multiple second feature blocks; each feature block pair includes a first feature block and a corresponding second feature block, each first feature block including N first feature points, each second feature block including N second feature points, where N is an integer greater than 1; for each feature block pair, determining the feature distance between each first feature point in the first feature block included in the feature block pair and each second feature point in the second feature block included in the feature block pair, and determining the reference feature point pair corresponding to the feature block pair based on the determined N×N feature distances; the feature distance is used to characterize the similarity between the first feature point and the second feature point. That is, the first feature map and the second feature map are divided into blocks, with a one-to-one correspondence between the first feature blocks and the second feature blocks, and a sub-pixel precision reference feature point pair is extracted for each feature block pair.

[0116] In some embodiments, determining the reference feature point pair corresponding to the feature block pair based on the determined N×N feature distances includes: determining a confidence matrix based on the N×N feature distances; the confidence matrix is ​​an N-row N-column matrix, and the element in the i-th row and j-th column of the confidence matrix is ​​used to characterize the feature distance between the i-th first feature point among the N first feature points and the j-th second feature point among the N second feature points, where i and j are integers less than or equal to N; determining the centroid position in the confidence matrix; and converting the centroid position into the coordinates of the first reference feature point and the second reference feature point in the reference feature point pair.

[0117] Optionally, a confidence matrix is ​​determined based on N×N feature distances, including: determining a mutual distance matrix based on N×N feature distances; the mutual distance matrix is ​​an N-row N-column matrix, and the element in the i-th row and j-th column of the mutual distance matrix is ​​the feature distance between the i-th first feature point among N first feature points and the j-th second feature point among N second feature points; and normalizing each element in the mutual distance matrix to obtain the confidence matrix.

[0118] It should be noted that the above implementation method is described using the determination process of a reference feature point pair as an example. In another embodiment, other reference feature point pairs can also be determined using the above implementation method, which will not be described in detail here.

[0119] See one example. Figure 4 The diagram shows a network structure where the first and second images are used as inputs to the target feature matching model. These images then pass through a first encoder to obtain a third and fourth feature map of H / 8×W / 8×64. Figure 4Taking one of the third and fourth feature maps as an example, then after the second constipation, the fifth and sixth feature maps of H / 8×W / 8×128 are obtained. Figure 4 Taking the fifth and sixth feature maps as an example, the third and fifth feature maps are then fused to obtain the first feature map H / 8×W / 8×128, and the fourth and sixth feature maps are then fused to obtain the second feature map H / 8×W / 8×128. Figure 4 (Taking one of the first and second feature maps as an example), then through the feature matching layer, a reference feature point pair is determined. Figure 4 (The example uses two reference feature point pairs). Finally, the reference feature point pairs are mapped to the first and second images to obtain the target feature point pairs.

[0120] See one example. Figure 5 The diagram shown illustrates the feature matching effect. Figure 5 It can be seen that many target feature point pairs can be identified for the first image and the second image, and the target feature point pairs are distributed in both the first image and the second image, showing excellent performance in both the textureless part and the repeated texture part of the first image and the second image.

[0121] See one example. Figure 6 The schematic diagram of the feature matching process is shown below. Taking any feature block pair as an example, and taking the size of the first feature block and the second feature block as k×k, the first feature block includes k×k first feature points, and the second feature block includes k×k second feature points. First, the first feature block and the second feature block are expanded. Then, the feature distance between the first first feature point and each second feature point is determined. Then, the feature distance between the second first feature point, the third first feature point, ... the k×kth first feature point and each second feature point is determined in sequence. Based on the determined feature distances, a confidence matrix of k×k rows and k×k columns is determined. Then, the centroid position (i, j) of the confidence matrix is ​​solved. Then, the centroid position is converted into the coordinates (x1, y1) of the first reference feature point and the coordinates (x2, y2) of the second reference feature point.

[0122] In some embodiments, a target feature matching model needs to be trained first. The training process of the target feature matching model includes: obtaining a training dataset; the training dataset includes multiple first sample images, second sample images corresponding to each first sample image, and multiple feature point pair labels, wherein the feature point pair labels include first sample feature points in the first sample images and second sample feature points corresponding to the first sample feature points in the second sample images corresponding to the first sample images; and training the feature matching model to be trained based on the training dataset to obtain the target feature matching model.

[0123] Optionally, acquiring a training dataset includes: acquiring a first depth image and a second depth image, wherein the first depth image and the second depth image are captured by two camera modules in the same depth camera on the same shooting scene; determining the correspondence between pixels in the first depth image and pixels in the second depth image; cropping the first depth image and the second depth image to obtain a first sample image and a second sample image; the baseline length between the first sample image and the second sample image conforms to the baseline length of the two camera modules in the electronic device; and based on the correspondence, determining the first sample feature points in the first sample image and the second sample feature points corresponding to the first sample feature points in the second sample image.

[0124] The depth camera belongs to a large baseline system, while the electronic device in this embodiment belongs to a small baseline system. The depth camera captures a third initial image and a fourth initial image. Then, based on its built-in software development kit, it acquires a first depth image corresponding to the third initial image and a second depth image corresponding to the fourth initial image. The first depth image and the second depth image record the distance information from each location point in the shooting scene to the depth camera.

[0125] Optionally, before determining the correspondence between pixels in the first depth image and pixels in the second depth image, the first depth image and the second depth image can be preprocessed, for example, by denoising and smoothing, to improve the accuracy of depth information in the processed first depth image and second depth image.

[0126] Of course, other methods can also be used to obtain the training dataset, and this disclosure does not limit the method of obtaining the training dataset.

[0127] It should be noted that the above embodiment only illustrates the example of calling the feature extraction network in the target feature matching model to extract the first feature map of the first image and the second feature map of the second image, and calling the feature matching layer in the target feature matching model to extract reference feature point pairs. In another embodiment, the target feature matching model may not be called, that is, the following steps are used for processing: feature extraction is performed on the first image to obtain the third feature map, feature extraction is performed on the second image to obtain the fourth feature map; feature extraction is performed on the third feature map to obtain the fifth feature map, feature extraction is performed on the fourth feature map to obtain the sixth feature map; the third and fifth feature maps are fused to obtain the first feature map, and the fourth and sixth feature maps are fused to obtain the second feature map; the first feature map is divided into multiple first feature blocks, the second feature map is divided into multiple second feature blocks, and multiple feature block pairs are determined based on the multiple first and multiple second feature blocks; for each feature block pair, the feature distance between each first feature point in the first feature block included in the feature block pair and each second feature point in the second feature block included in the feature block pair is determined, and the reference feature point pair corresponding to the feature block pair is determined based on the determined N×N feature distances.

[0128] In one example, the overall image processing flow from acquiring the first initial image and the second initial image to obtaining the corrected first target image and the second target image is as follows: Figure 7 As shown:

[0129] Figure 7 This is a flowchart illustrating an image processing method according to an exemplary embodiment, executed by an electronic device, see [link to flowchart]. Figure 7 The method includes the following steps:

[0130] Step S1: Acquire the first initial image and the second initial image.

[0131] Step S2: Preprocess the first initial image and the second image to obtain the first image and the second image.

[0132] Step S3: Perform feature matching on the first image and the second image to obtain target feature point pairs.

[0133] Step S4: Based on the target feature point pairs, iterate the initial intrinsic and extrinsic parameters of each camera module to obtain the target intrinsic and extrinsic parameters of each camera module.

[0134] Step S5: Based on the target intrinsic and target extrinsic parameters of each camera module, perform row alignment processing on the first image and the second image to obtain the first target image and the second target image.

[0135] The implementation methods for steps S1-S5 are the same as those in the above embodiments, and will not be repeated here.

[0136] In one example, a standard dataset is used to evaluate the image processing method provided in this disclosure embodiment. The evaluation metric is presented using the average row alignment error of the entire image pixels. Using traditional image correction methods, the average row alignment error is 0.8462 pixels, while using the image processing method provided in this disclosure embodiment, the average row alignment error is 0.2317 pixels. Compared with traditional image correction methods in related technologies, the image processing method provided in this disclosure embodiment has higher accuracy and better row alignment effect.

[0137] Figure 8 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment, configured in an electronic device, see [link to relevant documentation]. Figure 8 The device includes:

[0138] The initial image acquisition module 801 is configured to acquire a first initial image and a second initial image, which are obtained by two camera modules in the same electronic device capturing the same shooting scene.

[0139] The image preprocessing module 802 is configured to preprocess the first initial image and the second initial image to obtain a first image after processing the first initial image and a second image after processing the second initial image; the difference in viewpoint between the first image and the second image is less than a first threshold and the difference in scale is less than a second threshold.

[0140] The feature matching module 803 is configured to extract a first feature map of the first image and a second feature map of the second image, and to extract reference feature point pairs based on the first feature map and the second feature map; the reference feature point pairs include a first reference feature point in the first feature map and a second reference feature point in the second feature map that matches the first reference feature point;

[0141] The image correction module 804 is configured to correct the first image and the second image based on reference feature point pairs to obtain a first target image after the first image is corrected and a second target image after the second image is corrected.

[0142] In some embodiments, the image preprocessing module 802 is configured to:

[0143] The first initial image and the second initial image are cropped to obtain the first image and the second image.

[0144] In some embodiments, the image preprocessing module 802 is configured to:

[0145] Perform image quality correction on the first and second images to ensure that the image quality difference between the corrected first and second images is less than a third threshold; and / or,

[0146] The first image and the second image are subjected to distortion correction processing respectively to obtain the first image after distortion correction and the second image after distortion correction.

[0147] In some embodiments, the feature matching module 803 is configured to:

[0148] The target feature matching model is invoked to extract the first feature map of the first image and the second feature map of the second image, and reference feature point pairs are extracted based on the first and second feature maps; the target feature matching model is used to extract reference feature point pairs corresponding to the two input images.

[0149] In some embodiments, the target feature matching model includes a feature extraction network and a feature matching layer; the feature matching module 803 is configured to:

[0150] The feature extraction network is invoked to extract features from the first image and the second image respectively, resulting in a first feature map and a second feature map.

[0151] The feature matching layer is invoked to extract reference feature point pairs based on the first and second feature maps.

[0152] In some embodiments, the feature extraction network includes a first encoder, a second encoder, and a fusion layer; the feature matching module 803 is configured to:

[0153] The first encoder is invoked to extract features from the first image to obtain the third feature map, and the second image is extracted to obtain the fourth feature map; the size of the third feature map and the fourth feature map are both the first size, and the number of channels contained are both the first number;

[0154] The second encoder is invoked to extract features from the third feature map to obtain the fifth feature map, and to extract features from the fourth feature map to obtain the sixth feature map; the fifth and sixth feature maps are both of the first size, and both contain the second number of channels, which is greater than the first number;

[0155] The fusion layer is invoked to fuse the third and fifth feature maps to obtain the first feature map, and to fuse the fourth and sixth feature maps to obtain the second feature map. The first and second feature maps are both of the first size and contain the second number of channels.

[0156] In some embodiments, the feature matching module 803 is configured to:

[0157] The feature matching layer is invoked to perform the following steps:

[0158] The first feature map is divided into multiple first feature blocks, and the second feature map is divided into multiple second feature blocks. Based on the multiple first feature blocks and multiple second feature blocks, multiple feature block pairs are determined. Each feature block pair includes a first feature block and a second feature block corresponding to the first feature block. Each first feature block includes N first feature points, and each second feature block includes N second feature points, where N is an integer greater than 1.

[0159] For each feature block pair, the feature distance between each first feature point in the first feature block included in the feature block pair and each second feature point in the second feature block included in the feature block pair is determined. Based on the determined N×N feature distances, the reference feature point pair corresponding to the feature block pair is determined. The feature distance is used to characterize the similarity between the first feature point and the second feature point.

[0160] In some embodiments, the feature matching module 803 is configured to:

[0161] Based on N×N feature distances, a confidence matrix is ​​determined; the confidence matrix is ​​an N-row N-column matrix, and the element in the i-th row and j-th column of the confidence matrix is ​​used to characterize the feature distance between the i-th first feature point among N first feature points and the j-th second feature point among N second feature points, where i and j are integers less than or equal to N;

[0162] Determine the centroid location in the confidence matrix;

[0163] The centroid position is converted into the coordinates of the first reference feature point and the second reference feature point in the reference feature point pair.

[0164] In some embodiments, the feature matching module 803 is configured to:

[0165] Based on N×N feature distances, a mutual distance matrix is ​​determined; the mutual distance matrix is ​​an N-row N-column matrix, and the element in the i-th row and j-th column of the mutual distance matrix is ​​the feature distance between the i-th first feature point among N first feature points and the j-th second feature point among N second feature points;

[0166] Normalize each element in the mutual distance matrix to obtain the confidence matrix.

[0167] In some embodiments, the training module is configured as follows:

[0168] Obtain the training dataset; the training dataset includes multiple first sample images, a second sample image corresponding to each first sample image, and multiple feature point pair labels. The feature point pair labels include the first sample feature points in the first sample images and the second sample feature points in the second sample images corresponding to the first sample images.

[0169] Based on the training dataset, a feature matching model to be trained is trained to obtain a target feature matching model.

[0170] In some embodiments, the training module is configured as follows:

[0171] Acquire a first depth image and a second depth image, which are obtained by two camera modules in the same depth camera capturing the same shooting scene;

[0172] Determine the correspondence between pixels in the first depth image and pixels in the second depth image;

[0173] The first depth image and the second depth image are cropped to obtain a first sample image and a second sample image; the baseline length between the first sample image and the second sample image conforms to the baseline length of the two camera modules in the electronic device.

[0174] Based on the correspondence, the first sample feature points in the first sample image and the second sample feature points in the second sample image corresponding to the first sample feature points are determined.

[0175] In some embodiments, the image correction module 804 is configured to:

[0176] The reference feature point pair is mapped to the first image and the second image to obtain the target feature point pair; the target feature point pair includes the first target feature point in the first image and the second target feature point in the second image that matches the first target feature point.

[0177] Based on the target feature point pairs, the first image and the second image are corrected to obtain the first target image and the second target image.

[0178] In some embodiments, the image correction module 804 is configured to:

[0179] Using a pre-defined algorithm, based on the target feature point pairs and the initial intrinsic and extrinsic parameters of each camera module, the target intrinsic and extrinsic parameters of each camera module are determined.

[0180] Based on the target intrinsic and extrinsic parameters of each camera module, the first image and the second image are row aligned to obtain the first target image and the second target image.

[0181] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0182] This disclosure also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the image processing method described above.

[0183] Figure 9 This is a block diagram of an electronic device 900 according to an exemplary embodiment.

[0184] Reference Figure 9 The electronic device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0185] Processing component 902 typically controls the overall operation of electronic device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0186] Memory 904 is configured to store various types of data to support the operation of electronic device 900. Examples of this data include instructions for any application or method operating on electronic device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0187] Power supply component 906 provides power to various components of electronic device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 900.

[0188] Multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0189] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when electronic device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0190] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0191] Sensor assembly 914 includes one or more sensors for providing state assessments of various aspects of electronic device 900. For example, sensor assembly 914 can detect the on / off state of electronic device 900, the relative positioning of components such as the display and keypad of electronic device 900, changes in position of electronic device 900 or a component of electronic device 900, the presence or absence of user contact with electronic device 900, orientation or acceleration / deceleration of electronic device 900, and temperature changes of electronic device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0192] Communication component 916 is configured to facilitate wired or wireless communication between electronic device 900 and other devices. Electronic device 900 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0193] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0194] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of an electronic device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0195] This disclosure also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to perform the image processing method described above.

[0196] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0197] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first initial image and a second initial image; the first initial image and the second initial image are captured by two camera modules in the same electronic device on the same shooting scene; The first initial image and the second initial image are preprocessed to obtain a first image after processing the first initial image and a second image after processing the second initial image; the difference in viewpoint between the first image and the second image is less than a first threshold and the difference in scale is less than a second threshold. Extract a first feature map from the first image and a second feature map from the second image, and extract reference feature point pairs based on the first feature map and the second feature map; the reference feature point pairs include a first reference feature point in the first feature map and a second reference feature point in the second feature map that matches the first reference feature point; Based on the reference feature point pair, the first image and the second image are corrected to obtain a first target image after the first image is corrected and a second target image after the second image is corrected.

2. The image processing method according to claim 1, characterized in that, The preprocessing of the first initial image and the second initial image to obtain a first image after processing the first initial image and a second image after processing the second initial image includes: The first initial image and the second initial image are cropped to obtain the first image and the second image.

3. The image processing method according to claim 2, characterized in that, Before extracting the first feature map of the first image and the second feature map of the second image, the method further includes: Image quality correction is performed on the first image and the second image to ensure that the image quality difference between the corrected first image and the corrected second image is less than a third threshold; and / or, The first image and the second image are subjected to distortion correction processing respectively to obtain the first image after distortion correction and the second image after distortion correction.

4. The image processing method according to claim 1, characterized in that, The step of extracting a first feature map from the first image and a second feature map from the second image, and extracting reference feature point pairs based on the first feature map and the second feature map, includes: The target feature matching model is invoked to extract the first feature map of the first image and the second feature map of the second image, and the reference feature point pair is extracted based on the first feature map and the second feature map; the target feature matching model is used to extract the reference feature point pair corresponding to the two input images.

5. The image processing method according to claim 4, characterized in that, The target feature matching model includes a feature extraction network and a feature matching layer; the step of calling the target feature matching model to extract a first feature map of the first image and a second feature map of the second image, and extracting the reference feature point pair based on the first feature map and the second feature map, includes: The feature extraction network is invoked to extract features from the first image and the second image respectively, resulting in the first feature map and the second feature map. The feature matching layer is invoked to extract the reference feature point pair based on the first feature map and the second feature map.

6. The image processing method according to claim 5, characterized in that, The feature extraction network includes a first encoder, a second encoder, and a fusion layer; the step of calling the feature extraction network to extract features from the first image and the second image respectively to obtain the first feature map and the second feature map includes: The first encoder is invoked to extract features from the first image to obtain a third feature map, and features are extracted from the second image to obtain a fourth feature map; the size of the third feature map and the fourth feature map are both a first size, and the number of channels contained in both is a first number; The second encoder is invoked to extract features from the third feature map to obtain the fifth feature map, and to extract features from the fourth feature map to obtain the sixth feature map; the fifth feature map and the sixth feature map are both the first size, and both contain the second number of channels, wherein the second number is greater than the first number; The fusion layer is invoked to fuse the third feature map and the fifth feature map to obtain the first feature map, and to fuse the fourth feature map and the sixth feature map to obtain the second feature map; the size of the first feature map and the second feature map are both the first size, and the number of channels contained are both the second number.

7. The image processing method according to claim 5, characterized in that, The step of invoking the feature matching layer to extract reference feature point pairs based on the first feature map and the second feature map includes: The feature matching layer is invoked to perform the following steps: The first feature map is divided into multiple first feature blocks, and the second feature map is divided into multiple second feature blocks. Based on the multiple first feature blocks and the multiple second feature blocks, multiple feature block pairs are determined. Each feature block pair includes a first feature block and a second feature block corresponding to the first feature block. Each first feature block includes N first feature points, and each second feature block includes N second feature points, where N is an integer greater than 1. For each feature block pair, the feature distance between each first feature point in the first feature block included in the feature block pair and each second feature point in the second feature block included in the feature block pair is determined. Based on the determined N×N feature distances, a reference feature point pair corresponding to the feature block pair is determined. The feature distance is used to characterize the similarity between the first feature point and the second feature point.

8. The image processing method according to claim 7, characterized in that, The step of determining the reference feature point pair corresponding to the feature block pair based on the determined N×N feature distances includes: Based on the N×N feature distances, a confidence matrix is ​​determined; the confidence matrix is ​​an N-row N-column matrix, and the element in the i-th row and j-th column of the confidence matrix is ​​used to characterize the feature distance between the i-th first feature point among the N first feature points and the j-th second feature point among the N second feature points, where i and j are integers less than or equal to N; Determine the centroid location in the confidence matrix; The centroid position is converted into the coordinates of the first reference feature point and the second reference feature point in the reference feature point pair.

9. The image processing method according to claim 8, characterized in that, The process of determining the confidence matrix based on the N×N feature distances includes: Based on the N×N feature distances, a mutual distance matrix is ​​determined; the mutual distance matrix is ​​an N-row N-column matrix, and the element in the i-th row and j-th column of the mutual distance matrix is ​​the feature distance between the i-th first feature point among the N first feature points and the j-th second feature point among the N second feature points; The confidence matrix is ​​obtained by normalizing each element in the mutual distance matrix.

10. The image processing method according to any one of claims 4-9, characterized in that, The training process of the target feature matching model includes: Obtain a training dataset; the training dataset includes multiple first sample images, a second sample image corresponding to each first sample image, and multiple feature point pair labels, wherein the feature point pair labels include first sample feature points in the first sample images and second sample feature points in the second sample images corresponding to the first sample images that correspond to the first sample images; Based on the training dataset, a feature matching model to be trained is trained to obtain the target feature matching model.

11. The image processing method according to claim 10, characterized in that, The acquisition of the training dataset includes: A first depth image and a second depth image are acquired, wherein the first depth image and the second depth image are captured by two camera modules in the same depth camera on the same shooting scene; Determine the correspondence between pixels in the first depth image and pixels in the second depth image; The first depth image and the second depth image are cropped to obtain the first sample image and the second sample image; the baseline length between the first sample image and the second sample image conforms to the baseline length of the two camera modules in the electronic device. Based on the correspondence, the first sample feature point in the first sample image and the second sample feature point in the second sample image corresponding to the first sample feature point are determined.

12. The image processing method according to claim 1, characterized in that, The step of correcting the first image and the second image based on the reference feature point pair to obtain a first target image after correction of the first image and a second target image after correction of the second image includes: The reference feature point pair is mapped to the first image and the second image to obtain the target feature point pair; the target feature point pair includes a first target feature point in the first image and a second target feature point in the second image that matches the first target feature point. Based on the target feature point pair, the first image and the second image are corrected to obtain the first target image and the second target image.

13. The image processing method according to claim 12, characterized in that, The step of correcting the first image and the second image based on the target feature point pair to obtain the first target image and the second target image includes: Using a preset algorithm, based on the target feature point pairs and the initial intrinsic and extrinsic parameters of each camera module, the target intrinsic and extrinsic parameters of each camera module are determined; Based on the target intrinsic and target extrinsic parameters of each camera module, the first image and the second image are row aligned to obtain the first target image and the second target image.

14. An image processing apparatus, characterized in that, The device includes: The initial image acquisition module is configured to acquire a first initial image and a second initial image, wherein the first initial image and the second initial image are captured by two camera modules in the same electronic device on the same shooting scene; The image preprocessing module is configured to preprocess the first initial image and the second initial image to obtain a first image after processing the first initial image and a second image after processing the second initial image; the viewpoint difference between the first image and the second image is less than a first threshold and the scale difference is less than a second threshold. The feature matching module is configured to extract a first feature map of the first image and a second feature map of the second image, and extract reference feature point pairs based on the first feature map and the second feature map; the reference feature point pairs include a first reference feature point in the first feature map and a second reference feature point in the second feature map that matches the first reference feature point; The image correction module is configured to correct the first image and the second image based on the reference feature point pair to obtain a first target image after correction of the first image and a second target image after correction of the second image.

15. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the image processing method as described in any one of claims 1-13.

16. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image processing method as described in any one of claims 1-13.

Images