A method of image correction, storage medium and product

Abstract

Embodiments of the present application provide a kind of image correction method, storage medium and product, the method comprises: obtaining the image pair to be corrected, wherein the image pair to be corrected includes main camera image and secondary camera image;The image pair to be corrected is input target stereo rectification model and obtains secondary camera transformation matrix, wherein the target stereo rectification model at least includes target optical flow network, and the target optical flow network is used to extract matching attribute information from the image pair to be corrected;According to the secondary camera transformation matrix, the secondary camera image is transformed to obtain target image, and correction is completed. Through the technical scheme provided in the embodiments of the present application, the problem that calibration cannot be adapted to multi-camera system with zooming non-robustness can be solved, and the problem of non-robustness of weak texture and repeated texture scene for non-calibration can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular, the embodiments of the present application relate to a method for image correction, a storage medium and a product. BACKGROUND

[0002] Stereo rectification refers to performing a planar photography transformation on two images respectively, so that the corresponding epipolar lines of the two images are in the same horizontal direction, and the epipoles are mapped to the infinite distance. In this way, the two images only have a horizontal direction parallax, so that the stereo matching problem is reduced from two dimensions to one dimension, thereby improving the speed of image stereo matching. It can be understood that stereo rectification is an important method for improving the speed and accuracy of stereo matching.

[0003] The related art image stereo rectification method includes a stereo rectification method based on calibration. For example, the rectification algorithm proposed by Faugeras refers to re-projecting two images onto the same plane to obtain rectified images, and the re-projection plane passes through the intersection line of the two image planes and is parallel to the epipolar line. Fusiello proposed a simple image rectification method based on the Euclidean projection matrix. Meunier and Cox proposed a cylindrical projection algorithm in 1997, that is, the images are not projected onto a common plane, but onto a common cylindrical surface. The stereo rectification method based on calibration is a common solution at present, but it also has limitations. The stereo rectification method based on calibration can only be applied to a multi-camera system with a robust structure and including multiple cameras with fixed focal lengths. Since the multi-camera system included in a mobile phone and other related devices has a non-robust structure and is zoomable, the stereo rectification method based on calibration cannot be applied to devices with similar multi-camera systems.

[0004] Therefore, how to improve the stereo rectification effect of a mobile phone and other devices with similar multi-camera systems has become a technical problem to be solved. SUMMARY

[0005] The embodiments of the present application aim to provide a method for image correction, a storage medium and a product. The technical solutions provided by the embodiments of the present application can solve the problem that calibration cannot adapt to a multi-camera system with a non-robust zoom and improve the problem that non-calibration is not robust for weak texture and repetitive texture scenes.

[0006] In a first aspect, the embodiments of the present application provide a method for image correction, the method comprising: obtaining a pair of images to be corrected, wherein the pair of images to be corrected comprises a main camera image and a secondary camera image; inputting the pair of images to be corrected into a target stereo rectification model to obtain a secondary camera transformation matrix, wherein the target stereo rectification model at least comprises a target optical flow network, the target optical flow network is configured to extract matching attribute information from the pair of images to be corrected, and the secondary camera transformation matrix is obtained through the matching attribute information; and transforming the secondary camera image according to the secondary camera transformation matrix to obtain a target image, and completing the correction.

[0007] Some embodiments of the present application use an optical flow network to extract features to obtain a transformation matrix of a secondary camera image. The present application uses an optical flow network to obtain matching attribute information of left and right images (i.e., a main camera image and a secondary camera image). This can solve the problem that calibration cannot adapt to multi-camera systems with non-robust zooming, and can also improve the non-robust problem of weak texture and repetitive texture scenes in the calibration without calibration.

[0008] In some embodiments, the inputting the pair of images to be corrected into the target stereo rectification model to obtain the secondary camera transformation matrix comprises: inputting the pair of images to be corrected into the target stereo rectification model, obtaining an initial optical flow map through the target optical flow network in the target stereo rectification model, wherein the initial optical flow map is configured to represent position offset information between matching points on the main camera image and the secondary camera image; and obtaining the secondary camera transformation matrix according to the initial optical flow map.

[0009] In some embodiments, the target stereo rectification model further comprises a downsampling module configured to receive output data of the target optical flow network, and wherein the obtaining the secondary camera transformation matrix according to the initial optical flow map comprises: inputting the initial optical flow information into the downsampling module, and downsampling the initial optical flow information through the downsampling module to obtain a downsampling optical flow map; and obtaining the secondary camera transformation matrix according to the downsampling optical flow map.

[0010] In some embodiments, the target stereo rectification model further comprises a target depth convolutional network configured to receive output data of the downsampling module, and wherein the obtaining the secondary camera transformation matrix according to the downsampling optical flow map comprises: inputting the downsampling optical flow map into the target depth convolutional network, and obtaining the secondary camera transformation matrix through the target depth convolutional network.

[0011] Some embodiments of the present application use an optical flow network to obtain feature points, which can effectively overcome the defects of poor robustness of feature points obtained by non-calibration and poor weak texture processing effect. Moreover, some embodiments of the present application use a depth convolutional network to obtain fitting parameters, which is more stable than traditional parameter fitting methods.

[0012] After the secondary camera transformation matrix is obtained by inputting the image pair to be corrected into the target stereo rectification model, the method further includes: transforming the secondary camera image according to the secondary camera transformation matrix to obtain a target image, and completing stereo rectification.

[0013] Some embodiments of the present application use a secondary camera transformation matrix to perform coordinate transformation on a secondary camera image to obtain a stereo rectified image, which can improve the accuracy of further processing results such as image matching.

[0014] In some embodiments, before the secondary camera transformation matrix is obtained by inputting the image pair to be corrected into the target stereo rectification model, the method further includes: training a stereo rectification model according to training images to obtain the target stereo rectification model, wherein the training images include a plurality of groups of training image pairs, each group of training image pairs includes a primary camera sample image, a secondary camera sample image, and a target optical flow image, the target optical flow image is used to calculate a loss value, and the stereo rectification model includes an optical flow network and a deep convolutional network.

[0015] Some embodiments of the present application train a model based on training data to obtain a target stereo rectification model.

[0016] In some embodiments, training the stereo rectification model according to the training images includes: inputting the primary camera sample image and the secondary camera sample image into the optical flow network to obtain a predicted optical flow map; performing down-sampling processing on the predicted optical flow map to obtain a down-sampled predicted optical flow map; inputting the down-sampled predicted optical flow map into the deep convolutional network to obtain a predicted secondary camera transformation matrix; obtaining the loss value according to the target optical flow image and the predicted secondary camera transformation matrix, and adjusting parameters of the optical flow network and the deep convolutional network according to the loss value.

[0017] Some embodiments of the present application provide a method for training a stereo rectification model.

[0018] In some embodiments, obtaining the loss value according to the target optical flow image and the predicted secondary camera transformation matrix includes: obtaining a first loss value according to the target optical flow image and the predicted optical flow map; obtaining a second loss value through the predicted secondary camera transformation matrix, wherein the second loss value includes at least one of a third loss value for constraining an entire image and a fourth loss value for constraining sparse points, the entire image is obtained by the secondary camera sample image and the predicted secondary camera transformation matrix, and the sparse points are matching points corresponding to each grid point after down-sampling the primary camera sample image; and obtaining the loss value according to the first loss value and the second loss value.

[0019] The loss function designed by some embodiments of the present application at least includes a first loss value for reflecting the training of the optical flow network and a second loss value for reflecting the training of the deep convolutional network, and the second loss value evaluates the training of the deep convolutional network by at least one of constraints on the whole image and constraints on sparse points. It can be understood that the loss function of some embodiments of the present application can better adjust the parameters of the stereo rectification model in training to improve the effect of the target stereo rectification model obtained.

[0020] In some embodiments, the second loss value is obtained by the predicted secondary camera transformation matrix, comprising: obtaining a translation amount according to the predicted secondary camera transformation matrix; compensating the predicted secondary camera transformation matrix based on the translation amount to obtain a compensated predicted secondary camera transformation matrix; performing image affine transformation on the secondary camera sample image according to the compensated predicted secondary camera transformation matrix to obtain an updated secondary camera sample image; calculating the difference of pixel values of each pixel point on the aligned secondary camera sample image and the updated secondary camera sample image, and obtaining the third loss value based on the difference, wherein the aligned secondary camera sample image is an image after the stereo rectification algorithm.

[0021] Some embodiments of the present application provide a method for obtaining a translation amount and an updated secondary camera sample image according to the translation amount after compensating the predicted secondary camera transformation matrix, and then solving the third loss value. This method makes the calculation process of the constraint on the whole image more objective.

[0022] In some embodiments, the translation amount is obtained according to the predicted secondary camera transformation matrix, comprising: downsampling the target optical flow map to obtain a target size of the downsampled first target optical flow map; dividing the primary camera sample image into a plurality of image blocks to obtain a plurality of first grids, and determining the grid coordinates of each first grid in the plurality of first grids, wherein the total number of the plurality of first grids is the same as the number of pixels of the downsampled target optical flow map; obtaining the grid optical flow offset information of each first grid according to the downsampled first target optical flow map; summing up the grid coordinates of any first network and the grid optical flow offset information of the any first grid to obtain a matching point matched with the any first grid, and repeating the process to obtain the matching points of all first grids to obtain a matching point set; obtaining a transformed matching point set according to the matching point set and the predicted secondary camera transformation matrix, to obtain the transformed matching point coordinates corresponding to the matching points respectively; and obtaining the translation amount based on the grid coordinates of the any first grid and the transformed matching point coordinates corresponding to the any first grid.

[0023] Some embodiments of the present application provide a calculation method for obtaining a translation amount, which improves the objectivity and accuracy of the translation amount calculation.

[0024] In some embodiments, the second loss value is obtained by the predicted secondary camera transformation matrix, comprising: calculating a first item loss value, and calculating a second item loss value, wherein the first item loss value is used to constrain the row alignment, and the second item loss value is used to constrain the image translation amount; and obtaining the fourth loss value according to the first item loss value and the second item loss value.

[0025] Some embodiments of the present application define the fourth loss value for constraining the coefficient points by considering two aspects of constraining the row alignment and constraining the image translation amount, thereby improving the accuracy and objectivity of the fourth loss value.

[0026] In some embodiments, the first item loss value and the second item loss value are calculated, comprising: downsampling the predicted optical flow map to obtain a target size of a downsampled second predicted optical flow map; dividing the primary camera sample image into a plurality of image blocks to obtain a plurality of second grids, and determining the grid coordinates of each second grid in the plurality of second grids, wherein the total number of the plurality of second grids is the same as the number of pixels of the downsampled predicted optical flow map; obtaining the grid optical flow offset information of each second grid according to the downsampled second predicted optical flow map; solving the sum of the grid coordinates of any second network and the grid optical flow offset information of the any second grid to obtain a matching point matched with the any second grid, and repeating the process to obtain matching points of all second grids to obtain a secondary camera graph matching point set; obtaining a transformed secondary camera matching point set according to the secondary camera graph matching point set and the predicted secondary camera transformation matrix, wherein each element in the transformed secondary camera matching point set is used to represent the transformed coordinates of the matching point; and calculating the first item loss value and the second item loss value based on the grid coordinates of the any second grid and the transformed coordinates of the matching point corresponding to the any second grid.

[0027] Some embodiments of the present application provide a loss value calculation method for constraining sparse points, so that the fourth loss value is more objective and accurate.

[0028] In some embodiments, the first item loss value and the second item loss value are obtained based on the grid coordinates of the any second grid and the transformed coordinates of the matching point corresponding to the any second grid, comprising: calculating the first item loss value according to the longitudinal coordinate value of the grid coordinates and the longitudinal coordinate value of the transformed coordinates.

[0029] The row constraint of some embodiments of the present application is calculated by the longitudinal coordinate, thereby improving the objectivity and accuracy of the calculation of the first item loss value.

[0030] In some embodiments, the first loss value and the second loss value are obtained based on the transformed coordinates of the grid coordinates of any second grid and the matching points corresponding to the any second grid, including: solving the difference between the horizontal coordinate values of each grid coordinate corresponding to all second grids and the horizontal coordinate values of the corresponding transformed coordinates to obtain a plurality of difference values; and selecting the minimum value from the plurality of difference values as the second loss value.

[0031] Some embodiments of the present application can make the content at the farthest distance of the transformed secondary camera image closest to 0 by solving the minimum difference, that is, the minimum translation amount compared to the primary camera image, thereby improving the accuracy of the obtained second loss value.

[0032] In a second aspect, some embodiments of the present application provide an image correction device, including: a to-be-corrected image acquisition module configured to acquire a to-be-corrected image pair, wherein the to-be-corrected image pair includes a primary camera image and a secondary camera image; a processing module configured to input the to-be-corrected image pair into a target stereo rectification model to obtain a secondary camera transformation matrix, wherein the target stereo rectification model at least includes a target optical flow network configured to extract matching attribute information from the to-be-corrected image pair; and a transformation module configured to transform the secondary camera image according to the secondary camera transformation matrix to obtain a target image, thereby completing the correction.

[0033] In a third aspect, some embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, can implement the method according to any embodiment of the first aspect.

[0034] In a fourth aspect, some embodiments of the present application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method according to any embodiment of the first aspect.

[0035] In a fifth aspect, some embodiments of the present application provide a computer program product, including a computer program, wherein the computer program, when executed by a processor, can implement the method according to any embodiment of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0037] Figure 1 A comparison diagram of the main camera and the auxiliary camera before and after the stereo correction provided for the embodiments of the present application;

[0038] Figure 2 A comparison diagram of the main camera and the auxiliary camera before and after the stereo correction provided for the embodiments of the present application;

[0039] Figure 3 A schematic diagram of the image correction system provided for the embodiments of the present application;

[0040] Figure 4 A schematic diagram of the composition architecture of the target stereo correction model provided for the embodiments of the present application;

[0041] Figure 5 A flowchart of the image correction method provided for the embodiments of the present application;

[0042] Figure 6 A schematic diagram of the stereo correction model architecture during training provided for the embodiments of the present application;

[0043] Figure 7 A comparison diagram of the main camera and the auxiliary camera before and after the stereo correction provided for the embodiments of the present application; Figure 6 A training process flowchart for training the stereo model provided for the embodiments of the present application;

[0044] Figure 8 A composition block diagram of the image correction device provided for the embodiments of the present application;

[0045] Figure 9 A composition schematic diagram of the electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0047] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing description, and cannot be understood as indicating or implying relative importance.

[0048] Please refer to Figure 1 , Figure 1 A comparison diagram of the main camera and the auxiliary camera before and after the stereo correction provided for the embodiments of the present application;

[0049] Figure 1The example demonstrates the original poses of two cameras (i.e., the first camera 101 and the second camera 102) before stereo correction and the stereo-corrected state obtained after applying the stereo correction method to these two cameras. It can be understood that after stereo correction, the two cameras are ultimately positioned at the same horizontal level with the same pose and parallel optical axes. For example, as... Figure 2 As shown, the first image 201 (i.e., the main image) and the second image 202 (i.e., the secondary image to be corrected) are respectively obtained by using... Figure 1 The two original images were obtained by capturing images with the left and right cameras respectively. Figure 2 The third image 211 and the fourth image 212 are corrected images obtained by transforming the original image using the transformation matrix obtained through the stereo correction method. This can be understood as being based on... Figure 2 Stereo matching of the third image 211 (corresponding to the main image) and the fourth image 212 (the sub-image after the sub-image 202 has been corrected using the sub-image transformation matrix) can obviously improve the matching speed and accuracy.

[0050] Besides providing calibration-based stereo correction methods, related technologies also offer calibration-free stereo correction methods. For example, Pollefeys treats the poles as the origin of polar coordinates and uses coordinate transformation to correct image epipolar lines. Loop et al. divide the image correction process into two parts: projective transformation and ray transformation. However, projective transformation requires nonlinear solutions, and stability cannot be guaranteed. It should be noted that the inventors of this application discovered a calibration-free stereo correction scheme, which simplifies the calibration-based stereo correction scheme. However, it relies on the robustness of feature points extracted from the image, and therefore its stereo correction effect is poor for photographic systems with weak or repetitive textures.

[0051] To overcome the aforementioned problems, some embodiments of this application propose a stereo correction method based on an optical flow network (for feature extraction) and a deep convolutional network (for parameter fitting). The stereo correction method of this application can effectively improve the applicability of stereo correction and solve the problem of current uncalibrated schemes being insensitive to scene conditions. For example, the optical flow network can use a FlowNet2 network and a VGG network (as an example of a deep convolutional network) to fit the transformation matrix.

[0052] Since some embodiments of this application utilize the idea of ​​neural networks to replace traditional methods, it can not only solve the problem that calibration cannot be adapted to multi-camera systems with zoom inertia, but also improve the problem of uncalibrated systems being inert for weak and repetitive texture scenes.

[0053] The following is combined Figure 3It can be understood that some embodiments of the present application can also be applied to other devices with similar camera systems as the mobile phone to achieve stereo correction of the multiple cameras of these devices.

[0054] It should be noted that the output result of the stereo correction process of some embodiments of the present application is to obtain a secondary camera transformation matrix. It can be understood that the secondary camera transformation matrix is a transformation matrix for transforming the secondary camera image captured by the secondary camera. For example, since the angle between two or more cameras on the mobile phone is small, only the secondary camera transformation matrix can be obtained in the stereo correction process. It can be understood that if the distance between the multiple cameras on the device that needs to be stereo corrected is far (i.e., different from the characteristics of the multiple cameras of the mobile phone), in order to apply the technical solutions of the present application, it is also necessary to obtain the transformation matrix of the primary camera image in advance, and then transform the primary camera image captured by the primary camera according to the transformation matrix to obtain the primary camera image as the input of the target stereo correction network or as the training primary sample image to train the stereo correction network. It can be understood that how to obtain the transformation matrix of the primary camera image belongs to the common knowledge in the art, and will not be described in detail here.

[0055] Figure 3 The stereo correction system provided by some embodiments of the present application includes a mobile terminal 310 and a server 320, wherein, Figure 1 The mobile terminal 310 uses the primary camera 311 and the secondary camera 312 to capture two images of the same scene to obtain a pair of images to be corrected (i.e., the primary camera 311 captures a primary camera image 313, and the secondary camera 312 captures a secondary camera image 314), and sends the two captured images to the server 320. The server 320 is configured to obtain a transformation matrix according to the two images and to transform the secondary camera image according to the transformation matrix to obtain a corrected target secondary camera image.

[0056] For example, in some embodiments of the present application, the primary camera 311 and the secondary camera 312 are both RGB cameras.

[0057] The mobile terminal 310 can also be a PAD, and the present application does not limit the specific type of terminal. The mobile terminal 310 and the server 320 can be interconnected through a wireless network or a wired network to realize the transmission of information (e.g., obtaining the first frame image, the second frame image, and the target registration image) between the two.

[0058] It should be noted that in some embodiments of the present application, the functions of the server 320 can also be performed by the mobile terminal 310, that is, the terminal 310 obtains the transformation matrix according to the two images and transforms the secondary camera image according to the transformation matrix to obtain the corrected target secondary camera image. In these embodiments, there is no need to arrange the server 320. Figure 3

[0059] The stereoscopic correction method performed by the mobile terminal 10 or the server provided by some embodiments of the present application will be described below in conjunction with examples. Figure 4

[0060] As shown in the figure, the method for image correction provided by the embodiments of the present application comprises the following steps. Figure 4 S101, obtaining a pair of images to be corrected, wherein the pair of images to be corrected comprises a primary camera image and a secondary camera image; S102, inputting the pair of images to be corrected into a target stereoscopic correction model to obtain a secondary camera transformation matrix, wherein the target stereoscopic correction model at least comprises a target optical flow network, the target optical flow network is used to extract matching attribute information from the pair of images to be corrected, and the secondary camera transformation matrix is obtained through the matching attribute information; S103, transforming the secondary camera image according to the secondary camera transformation matrix to obtain a target image, and completing correction.

[0061] It should be noted that some embodiments of the present application use an optical flow network to extract features to obtain the transformation matrix of the secondary camera image. The present application uses the optical flow network to obtain the matching information of the left and right images (i.e. the primary camera image and the secondary camera image), which can solve the problem that the calibration cannot adapt to the multi-camera system with non-robust zooming, and can also improve the problem that the non-calibration is not robust to weak texture and repetitive texture scenes.

[0062] The target stereoscopic correction model provided by some embodiments of the present application will be described below in conjunction with examples. Figure 5

[0063] As shown in the figure, the target stereoscopic correction model 200 provided by some embodiments of the present application comprises a target optical flow network 131, a down-sampling processing module 102 and a target deep convolution network 133. Figure 5 For example, the target optical flow network 131 can be a FlowNet2 network, and can also be other types of optical flow networks. The target deep convolution network 133 can adopt a VGG network. The down-sampling processing module 102 can be implemented by a pooling layer, for example, the pooling layer can be a mean pooling or a max pooling.

[0064]

[0065] Figure 5 ​​​​The target stereo rectification model is input with the main camera image 111 and the auxiliary camera image 112. It can be understood that the two images are the image pair to be rectified this time. Figure 5 In addition to the target stereo rectification model, a transformation module is also included. It can be understood that the auxiliary camera image 112 can be coordinate-transformed based on the target auxiliary camera transformation matrix 141 through the transformation module to obtain the target auxiliary camera image 151. The main camera image 111 and the auxiliary camera image 112 can have Figure 2 the above two images have problems, but the main camera image 111 and the target auxiliary camera image 151 will have effects similar to Figure 2 the lower two images, that is, the same photographed object point is located on the same horizontal line in the two images, and only the horizontal coordinate difference exists.

[0066] The stereo rectification process of the present application will be described below. Figure 5

[0067] For example, in some embodiments of the present application, the above S102 exemplarily includes: inputting the image pair to be rectified into the target optical flow network, and obtaining an initial optical flow map through the target optical flow network, wherein the initial optical flow map is used to represent the positional offset information between the matching points on the main camera image and the auxiliary camera image; obtaining an auxiliary camera transformation matrix according to the initial optical flow information. For example, the target stereo rectification model 200 includes a downsampling module, and the process of obtaining the auxiliary camera transformation matrix according to the initial optical flow information exemplarily includes: inputting the initial optical flow information into the downsampling module, and obtaining a down-sampled optical flow map through the downsampling module, and obtaining the auxiliary camera transformation matrix according to the down-sampled optical flow map. For example, the target stereo rectification model includes a target depth convolution network, and the process of obtaining the auxiliary camera transformation matrix according to the down-sampled optical flow map exemplarily includes: inputting the down-sampled optical flow map into the target depth convolution network, and obtaining the auxiliary camera transformation matrix through the target depth convolution network.

[0068] Some embodiments of the present application use the optical flow network to obtain the feature points, which can effectively overcome the defects of poor robustness of feature points obtained by the non-calibration method and poor processing effect for weak texture. Moreover, the depth convolution network used in some embodiments of the present application to obtain the fitting parameters is more stable in technical effect compared with the traditional parameter fitting method. Some embodiments of the present application use the pooling layer to down-sample the output of the optical flow network to adapt to the size requirements of the input image of the target depth convolution network.

[0069] It can be understood that Figure 5 The target stereo rectification model 200 is a network model trained by training data. The process of training the stereo rectification model to obtain the target stereo rectification model will be described below. ​

[0070] In other words, in some embodiments of this application, before executing S101, the image correction method further includes: training a stereo correction model based on training images to obtain the target stereo correction model, wherein the training images include multiple sets of training image pairs, each set of training image pairs including: a main camera sample image, a secondary camera sample image, and a target optical flow image, the target optical flow image being used to calculate the loss value, and the stereo correction model including an optical flow network and a deep convolutional network. Some embodiments of this application train the model based on training data to obtain the target stereo correction model.

[0071] The following example illustrates a training process using secondary camera sample images and primary camera sample images. Figure 6 As shown, the stereo correction model 100 includes an optical flow network 171, a downsampling processing module 102, and a deep convolutional network 103. The training images include a main camera sample image 111, a secondary camera sample image 112, an ideal optical flow image 113 (or target optical flow map), and an aligned secondary camera sample image 114. The stereo correction model outputs the predicted secondary camera transformation matrix 121 during training, which is subsequently used to calculate the loss value. It is understood that during the training process, in order to determine the direction of parameter adjustment, it is necessary to pre-acquire the target optical flow image (i.e., the ideal optical flow map) and the aligned secondary camera sample image (i.e., the secondary camera sample image aligned with the main camera sample image obtained through existing algorithms). These images can be used to calculate the loss value, thereby determining the direction of parameter adjustment.

[0072] For example, in some embodiments of this application, training the stereo correction model based on the training images to obtain the target stereo correction model includes:

[0073] S201, input the main camera sample image and the secondary camera sample image into the optical flow network to obtain the predicted optical flow map.

[0074] Understandably, the main camera sample image and the secondary camera sample image are images obtained by capturing the same scene simultaneously from the main camera and the secondary camera. Furthermore, to determine the direction of parameter adjustment, it is necessary to pre-acquire an aligned secondary camera sample image that is aligned with the main camera sample image, as well as a target optical flow image to verify the quality of the predicted optical flow image. For example, the methods for acquiring the aligned secondary camera sample image include: using a robust binocular structure to perform stereo correction on the acquired images through accurate calibration data, or constructing a perfect parallel binocular camera using simulation software to acquire binocular images of a virtual scene.

[0075] S202, the predicted optical flow map is downsampled to obtain a downsampled predicted optical flow map.

[0076] For example, the predicted optical flow image is subjected to a uniform pooling operation to obtain a down-sampled predicted optical flow image.

[0077] S203, inputting the down-sampled predicted optical flow image into the deep convolutional network to obtain a predicted secondary camera transformation matrix.

[0078] For example, the predicted secondary camera transformation matrix can be a 3*3 matrix.

[0079] S204, obtaining the loss value according to the target optical flow image and the predicted secondary camera transformation matrix, and adjusting the parameters of the optical flow network and the deep convolutional network according to the loss value.

[0080] The implementation process of the fourth step is exemplarily described below.

[0081] It can be understood that the loss value constructed by some embodiments of the present application needs to be able to reflect the loss value of the training of the optical flow network and also needs to be able to reflect the loss value of the quality of the predicted secondary camera transformation matrix. For example, in some embodiments of the present application, the above-mentioned S204 described as obtaining the loss value according to the target optical flow image and the predicted secondary camera transformation matrix exemplarily includes the following three sub-steps:

[0082] The first sub-step is to obtain a first loss value according to the target optical flow image and the predicted optical flow image.

[0083] For example, the first loss value is used to constrain the optical flow offset of each pixel point on the target optical flow image and the predicted optical flow image.

[0084] The second sub-step is to obtain a second loss value through the predicted secondary camera transformation matrix, wherein the second loss value includes at least one of a third loss value used to constrain an entire image and a fourth loss value used to constrain sparse points, the entire image is obtained through the secondary camera sample image and the predicted secondary camera transformation matrix, and the sparse points are matching points corresponding to each grid point after the down-sampling of the primary camera sample image.

[0085] That is, in some embodiments of the present application, the second loss value is the third loss value. In some other embodiments of the present application, the second loss value is the fourth loss value. In some other embodiments of the present application, the second loss value includes both the third loss value and the fourth loss value, for example, the second loss value is the sum of the third loss value and the fourth loss value.

[0086] The process of obtaining the third loss value is exemplarily described below.

[0087] In some embodiments of the present application, the second sub-step example comprises: obtaining a translation amount according to the predicted secondary camera transformation matrix, wherein the translation amount is used to represent the predicted secondary camera transformation matrix and true; compensating the predicted secondary camera transformation matrix based on the translation amount to obtain a compensated predicted secondary camera transformation matrix; performing image affine transformation on the secondary camera sample image according to the compensated predicted secondary camera transformation matrix to obtain an updated secondary camera sample image; calculating the difference of pixel values of each pixel point on the aligned secondary camera sample image and the updated secondary camera sample image, and obtaining the third loss value based on the difference, wherein the aligned secondary camera sample image is an image after a stereo rectification algorithm.

[0088] That is, some embodiments of the present application provide a method of obtaining a translation amount and an updated secondary camera sample image according to the translation amount after compensating the predicted secondary camera transformation matrix, and then solving the third loss value. This method makes the calculation process of constraining the entire image more objective.

[0089] In order to obtain the translation amount, in some embodiments of the present application, the obtaining of the translation amount according to the predicted secondary camera transformation matrix comprises: downsampling the target optical flow map to obtain a target size of a downsampled first target optical flow map; dividing the primary camera sample image into a plurality of image blocks to obtain a plurality of first grids, and determining the grid coordinates of each first grid in the plurality of first grids, wherein the total number of the plurality of first grids is the same as the number of pixels of the downsampled target optical flow map; obtaining grid optical flow offset information of each first grid according to the downsampled first target optical flow map; summing up the grid coordinates of any first network and the grid optical flow offset information of the any first grid to obtain a matching point matched with the any first grid, repeating the process to obtain matching points of all first grids to obtain a matching point set; obtaining a transformed matching point set according to the matching point set and the predicted secondary camera transformation matrix, obtaining a transformed matching point coordinate corresponding to each matching point; and obtaining the translation amount based on the grid coordinates of the any first grid and the transformed matching point coordinates corresponding to the any first grid. Some embodiments of the present application provide a calculation method for obtaining a translation amount, which improves the objectivity and accuracy of the translation amount calculation.

[0090] The implementation process of obtaining the fourth loss value of S204 is described below.

[0091] In some embodiments of the present application, the second sub-step example comprises: calculating a first loss value and calculating a second loss value, wherein the first loss value is used to constrain the row alignment, and the second loss value is used to constrain the image translation amount; and obtaining the fourth loss value according to the first loss value and the second loss value. For example, in some embodiments of the present application, the fourth loss value is obtained by directly summing the first loss value and the second loss value, and in some embodiments of the present application, the fourth loss value can also be obtained by weighted summing the first loss value and the second loss value, etc.

[0092] It should be noted that the algorithm for obtaining the first loss value can include: RMSE (Root Mean Square Error), MSE (Mean Square Error), MAE (Mean Absolute Error), and SD (Standard Deviation).

[0093] That is, some embodiments of the present application define the fourth loss value for constraining the coefficient points by considering two aspects of constraining the row alignment and constraining the image translation amount, thereby improving the accuracy and objectivity of the fourth loss value.

[0094] For example, in some embodiments of the present application, the calculation of the first loss value and the calculation of the second loss value example comprises: downsampling the predicted optical flow map to obtain a target size of a downsampled second predicted optical flow map; dividing the main camera sample image into a plurality of image blocks to obtain a plurality of second grids, and determining the grid coordinates of each second grid in the plurality of second grids, wherein the total number of the plurality of second grids is the same as the number of pixels of the downsampled predicted optical flow map; obtaining the grid optical flow offset information of each second grid according to the downsampled second predicted optical flow map; solving the sum of the grid coordinates of any second network and the grid optical flow offset information of the any second grid to obtain a matching point matched with the any second grid, and repeating the process to obtain matching points of all second grids to obtain a secondary camera graph matching point set; obtaining a transformed secondary camera matching point set according to the secondary camera graph matching point set and the predicted secondary camera transformation matrix, wherein each element in the transformed secondary camera matching point set is used to represent the transformed coordinates of the matching point; and calculating the first loss value and the second loss value based on the grid coordinates of the any second grid and the transformed coordinates of the matching point corresponding to the any second grid. Some embodiments of the present application provide a loss value calculation method for constraining sparse points, so that the fourth loss value is more objective and accurate.

[0095] It should be noted that some embodiments of the present application obtain the first loss value and the second loss value according to the grid coordinates and the transformed coordinates.

[0096] For example, in some embodiments of the present application, the first loss value and the second loss value are obtained based on the grid coordinates of any of the second grids and the transformed coordinates of the matching points corresponding to the any of the second grids, including: calculating the first loss value according to the longitudinal coordinate values of the grid coordinates and the longitudinal coordinate values of the transformed coordinates. The row constraint in some embodiments of the present application is calculated by the longitudinal coordinate, which improves the objectivity and accuracy of the calculation of the first loss value.

[0097] For example, in some embodiments of the present application, the first loss value and the second loss value are obtained based on the grid coordinates of any of the second grids and the transformed coordinates of the matching points corresponding to the any of the second grids, including: calculating the first loss value according to the longitudinal coordinate values of the grid coordinates and the longitudinal coordinate values of the transformed coordinates. The row constraint in some embodiments of the present application is calculated by the longitudinal coordinate, which improves the objectivity and accuracy of the calculation of the first loss value.

[0098] Third sub-step, obtaining the loss value according to the first loss value and the second loss value.

[0099] For example, in some embodiments of the present application, the loss value is obtained by directly summing the first loss value and the second loss value. In some other embodiments of the present application, the loss value is obtained by weighted summing the first loss value and the second loss value.

[0100] The loss function designed in some embodiments of the present application at least includes the first loss value for reflecting the training situation of the optical flow network and the second loss value for reflecting the training situation of the depth convolutional network, and the second loss value adopts at least one of the constraints on the whole image and the constraints on the sparse points to evaluate the training situation of the depth convolutional network. It can be understood that the loss function in some embodiments of the present application can better adjust the parameters of the stereo rectification model in training and improve the effect of the obtained target stereo rectification model.

[0101] The following will be illustrated by taking two specific size left images I L (as an example of a main camera sample image), right images I R (as an example of a sub-camera sample image), optical flow GT images I G (as an example of a target optical flow image), and aligned GT right images I R_G (as an example of an aligned sub-camera sample image) as examples to illustrate the training process. The left images I L , the right images I R , the optical flow GT images I GAlign GT right image I R_G The stereo rectification model is trained. Specifically, the training process exemplarily includes:

[0102] In the first step, the left image I L , the right image I R and the optical flow GT image I G are input to the FlowNet2 network (as an example of an optical flow network), and the purpose of using the network is to obtain the matching information of the left image and the right image. The reason why the CNN feature and other feature networks are not used here is that the optical flow network has better performance in the uniformity of features and weak texture scenes.

[0103] The size and channel number of the left image I L and the right image I R are represented as 512x384x3, and the predicted optical flow image output by the FlowNet2 network is 512x384x2. The optical flow Figure 1 is generally 2 channels, one channel stores the x direction offset, and the other channel stores the y direction offset.

[0104] In the second step, a 2x2 pooling layer (as an example of a down-sampling module) is used to sample the predicted optical flow image, and the optical flow image obtained after sampling is a 256x192x2 optical flow image.

[0105] It should be noted that the optical flow is the positional offset relationship between the two images, for example, the position of P 1L in the left image is P 1R in the right image, so the optical flow of the P 1L point is (x2-x1, y2-y1), and the optical flow image is a set of positional offsets of each pixel position in the whole image.

[0106] In the third step, the optical flow image obtained in the second step is input to the VGG network (as an example of a deep convolutional network), and the size of the image is 256x192x2. The VGG network outputs a 1x8 vector. The parameters of the VGG network used in this example are: the first layer is 256x192x48, the second layer is 128x96x96, the third layer is 64x48x192, the fourth layer is 32x24x384, the fifth layer is 16x12x384, the sixth layer is 8x6x384, the seventh layer is 1x1x1028, and the eighth layer is 1x1x8.

[0107] In the fourth step, the matrix for transforming the right image is constructed according to the vector output by the VGG network. The predicted affine transformation matrix is obtained.

[0108] The process of obtaining the fourth loss value is exemplarily described below.

[0109] 4.1, the predicted optical flow map obtained in the first step (or the optical flow map obtained in the second step) is sampled to obtain a two-channel optical flow map with a size of 32x24 (an example of a target size), a down-sampled second predicted optical flow map of the target size is obtained, and the down-sampled second predicted optical flow map is combined with the image grid coordinates to obtain 768 (calculated by 32x24) matching point pairs P L and P R . The right image points are transformed to obtain a transformed secondary camera matching point set P RN = H*P R , and then an example of a calculation formula of the fourth loss value is: Loss1 = RMSE(P L -P RN )| y + MIN(P L -P RN )| x , where RMSE(P L -P RN )| y represents taking the y-coordinate constraint row alignment, and MIN(P L -P RN )| x represents taking the x-coordinate minimum constraint to limit the translation amount of the image.

[0110] Down-sampling the predicted optical flow map to 32x24x2 can reduce the amount of calculation and the amount of parameters, and prevent the memory of the VGG network from exploding.

[0111] The above 4.1 also includes the following implementation process:

[0112] The method for obtaining the matching point pairs P L and P R (P L and P R are two-dimensional coordinates of N*2, N is a constant and its value is the number of sampling grid points) is as follows:

[0113] The optical flow offset of the obtained down-sampled second predicted optical flow map is the optical flow offset corresponding to each grid point when the predicted optical flow map (single-channel size is 512x384) is uniformly divided into 32x24 grids. After the primary camera sample image (original image size is 512x384) is uniformly divided into 32x24 (an example of a target size) grids, 32x24 image blocks are obtained, each image block corresponds to a second grid, and then the coordinates of each second grid point are exemplified as: (512 / 32*i, 384 / 24*j) i=0,1,..31, j=0,1,…,23.

[0114] The coordinate points of the left image (as the coordinate points corresponding to each second grid) P L= (512 / 32*i, 384 / 24*j) i=0,1,..31, j=0,1,..,23, the optical flow corresponding to each coordinate point of each second grid can be obtained from the 32x24x2 optical flow map, and the corresponding coordinate point (i.e., each matching point in the secondary camera matching point set) P R = P L + flow(x,y)*2 (i.e., by solving the grid coordinate P L of any second grid and summing the grid optical flow offset information (i.e., flow(x,y)*2) of the any second grid, a matching point matched with the any second grid is obtained (i.e., P R is obtained), where flow(x,y) is the (x,y) offset obtained from the 32x24x2 optical flow map, and flow(x,y)*2 is because the scale of the 32x24x2 optical flow map is 256x192x2, which is 1 / 2 of the original map.

[0115] It can be understood that P L is a 32*24 grid coordinate point, and P R is the grid point on the left plus the optical flow offset to obtain the grid coordinate point on the right.

[0116] The coordinate conversion of the above 4.1 is P RN = H*P R , where H is a 3x3 matrix, P R is the Nx2 two-dimensional coordinate conversion to 3xN secondary coordinates, and the two-dimensional coordinate represents P=(x,y) corresponding to the secondary coordinates

[0117] The calculation formula of the above first loss value is n, and the value range of n is greater than or equal to 1 and less than or equal to N:

[0118]

[0119] The second loss value calculation formula is:

[0120] MIN(P L -P RN )| x = MIN(P L.x0 -PR N.x0 , P L.x1 -PR N.x1 , …, P L.xn -P RN.xn ),

[0121] Where,.x represents taking x coordinate,.y represents taking y coordinate, x0, x1…xn represent horizontal coordinate values, of course, RMSE can also be replaced with MSE, L1 and other simple loss, and the result will exist certain fluctuation. The first loss value involves the RMSE (root mean square error) algorithm, which can be replaced by: MSE (mean square error), MAE (mean absolute error) or SD (standard deviation).

[0122] The RMSE (P L RN )| y is used to constrain the y coordinate, and the sub-camera transformation matrix H is expected to be obtained, so that the content in the transformed sub-camera image is on the same line as the corresponding content of the main camera. MIN (P L RN )| x is used to constrain the translation of the sub-camera image, and the sub-camera transformation matrix H is expected to be obtained, so that the content at the farthest place in the transformed sub-camera image has the smallest translation compared with the main camera, which tends to be close to 0.

[0123] The process of obtaining the third loss value is described below.

[0124] 4.2 uses P RN | x in 4.1 to calculate the GT optical flow I G After the change (4.1 operation) P R_G_RN | x , the difference between the predicted H and the true value (translation) t = P RN | x R_G_RN | x is calculated. Then the predicted H is compensated for translation The input right image I RN is transformed to I R , so Loss2 = RMSE (I RN -I R_G ) (as an example of the third loss value).

[0125] The transformation matching point set P R_G_RN in the above 4.2 is obtained in the following way:

[0126] The GT image I G (512x384x2) is down-sampled to 32x24x2 to obtain the target size of the down-sampled first target optical flow image. The main camera sample image (i.e. the left image with a size of 512x384) is uniformly divided into 32x24 image blocks to obtain 32x24 first grids, and the grid coordinates P L_G ​​​= (512 / 32*i, 384 / 24*j)i=0,1,..31, j=0,1,..23. The optical flow corresponding to each coordinate point of the first grid can be obtained from the 32x24x2 I G The grid optical flow offset information of each first grid is obtained (i.e. the grid optical flow offset information of each first grid is obtained according to the down-sampled first target optical flow map). The corresponding coordinate point (i.e. the matching point matched with the any first grid) P R_G = P L_G + flow(x,y), where flow(x,y) does not need to be multiplied by a factor, because the optical flow map is down-sampled at the original size, thus obtaining the accurate two-dimensional matching point pair P L_G and P R_G , the right Figure 2 dimensional point is transformed P R_G_RN = H*P R_G , to obtain the transformed matching point set.

[0127] The above translation amount calculation formula is exemplified as:

[0128]

[0129] That is, in some embodiments of the present application, the position of the translation amount is the average value of all matching point x coordinate difference values.

[0130] The above transformation of the input right image I RN = H*I R is the warp transformation of the image.

[0131] The above third loss value can be exemplified as:

[0132]

[0133] The third loss value expression is used to represent the constraint on the color value of each pixel on the image. The RMSE (Root Mean Square Error) algorithm involved in the third loss value can be replaced by: MSE (Mean Square Error), MAE (Mean Absolute Error) or SD (Standard Deviation).

[0134] 4.3 First loss value exemplified calculation method

[0135] For the predicted optical flow map I P out of the optical flow network, it needs to be compared with the GT, and the first loss value is exemplified as:

[0136]

[0137] The first loss value represents a constraint on the optical flow offset of each pixel of the image. The RMSE (Root Mean Square Error) algorithm involved in the first loss value can be replaced by: MSE (Mean Square Error), MAE (Mean Absolute Error), or SD (Standard Deviation).

[0138] As an example of the present application, the loss function value is the sum of the first loss value, the third loss value and the fourth loss value, which is expressed by the formula: Loss = Loss1 + Loss2 + Loss3.

[0139] Please refer to Figure 8 , Figure 8 The device for image correction provided by the embodiments of the present application is shown, and it should be understood that the device corresponds to the above-mentioned Figure 4 method embodiments, and can perform each step involved in the above-mentioned method embodiments. The specific functions of the device can be referred to the description in the foregoing, and the detailed description is appropriately omitted here to avoid repetition. The device includes at least one software function module stored in the form of software or firmware in the memory or solidified in the operating system of the device. The device for image correction includes: a to-be-corrected image acquisition module 801, a processing module 802, and a transformation module 803.

[0140] The to-be-corrected image acquisition module 801 is configured to acquire a to-be-corrected image pair, wherein the to-be-corrected image pair includes a main camera image and a sub-camera image.

[0141] The processing module 802 is configured to input the to-be-corrected image pair into a target stereo rectification model to obtain a sub-camera transformation matrix, wherein the target stereo rectification model at least includes a target optical flow network, and the target optical flow network is used to extract matching information from the to-be-corrected image pair.

[0142] The transformation module is configured to transform the sub-camera image according to the sub-camera transformation matrix to obtain a target image, and complete stereo rectification.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the foregoing method, and will not be described in more detail here.

[0144] Some embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method described in any of the above embodiments of the image correction method.

[0145] Some embodiments of the present application provide a computer program product, which includes a computer program, wherein the computer program is executed by a processor to implement the method described in any of the above embodiments of the image correction method.

[0146] As Figure 9 shown in FIG. 9, some embodiments of the present application provide an electronic device 900, comprising a memory 910, a processor 920, and a computer program stored in the memory 910 and capable of running on the processor 920, wherein the processor 920 reads the program from the memory 910 through a bus 930 and implements the method described in any of the embodiments of the image correction method above when executing the program.

[0147] The processor 520 can process digital signals and can include various computing structures. For example, a complex instruction set computer structure, a reduced instruction set computer structure, or a structure implementing a combination of multiple instruction sets. In some examples, the processor 520 can be a microprocessor.

[0148] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of the instructions. These instructions and / or data can include code for implementing some or all of the functions of one or more modules described in the embodiments of the present application. The processor 520 of the embodiments of the present disclosure can be used to execute the instructions in the memory 510 to implement the method shown in FIG. 5. The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memories well known to those skilled in the art. Figure 4

[0149] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other means. The apparatus embodiments described above are only schematic; for example, the flowcharts and block diagrams in the drawings show possible implementation architectures, functions, and operations of the apparatus, method, and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks shown in succession can in fact be executed substantially concurrently, or the blocks can sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each block and combination of blocks in the block diagrams and / or flowcharts can be implemented by special-purpose hardware-based systems that perform the specified functions or acts, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0150] ​In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0151] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0152] The above is only an embodiment of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0153] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the present application, which shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0154] It is to be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a combination of two or more components. Additionally, the terms "comprise," "comprises," and "comprising," or any variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless otherwise indicated herein, the terms "first," "second," "third," etc., are used herein merely as labels, and are not intended to impose ordinal import.

Claims

1. A method for image correction, characterized in that, The method includes: Obtain a pair of images to be corrected, wherein the pair of images to be corrected includes a main camera image and a secondary camera image; The image pair to be corrected is input into the target stereo correction model to obtain the sub-camera transformation matrix, wherein the target stereo correction model includes at least a target optical flow network, the target optical flow network is used to extract matching attribute information from the image pair to be corrected, and the sub-camera transformation matrix is ​​obtained through the matching attribute information; The target image is obtained by transforming the secondary camera image according to the secondary camera transformation matrix, thus completing the correction. The step of obtaining the sub-camera transformation matrix by applying the image to be corrected to the input target stereo correction model includes: The image to be corrected is input into the target stereo correction model, and an initial optical flow map is obtained through the target optical flow network in the target stereo correction model. The initial optical flow map is used to characterize the positional offset information between matching points on the main image and the sub-image. The sub-percussion transformation matrix is ​​obtained based on the initial optical flow diagram.

2. The method as described in claim 1, characterized in that, The target stereo correction model further includes: a downsampling module, wherein... The step of obtaining the sub-percussion transformation matrix based on the initial optical flow map includes: The initial optical flow map is input into the downsampling module, and the initial optical flow map is downsampled by the downsampling module to obtain a downsampled optical flow map; The sub-percussion transformation matrix is ​​obtained based on the downsampled optical flow map.

3. The method as described in claim 2, characterized in that, The target stereo correction model further includes: a target depthwise convolutional network, wherein... The step of obtaining the sub-percussion transformation matrix based on the downsampled optical flow map includes: The downsampled optical flow map is input into the target depth convolutional network, and the sub-camera transformation matrix is ​​obtained through the target depth convolutional network.

4. The method according to any one of claims 1-3, characterized in that, Before obtaining the subcamera transformation matrix from the image to be corrected using the input target stereo correction model, the method further includes: Input the main camera sample image and the secondary camera sample image into the optical flow network to obtain the predicted optical flow map; The predicted optical flow map is downsampled to obtain a downsampled predicted optical flow map; The downsampled predicted optical flow map is input into a deep convolutional network to obtain the predicted sub-sensor transformation matrix; The loss value is obtained based on the target optical flow map and the predicted sub-sensor transformation matrix, and the parameters of the optical flow network and the deep convolutional network are adjusted based on the loss value to obtain the target stereo correction model.

5. The method as described in claim 4, characterized in that, The step of obtaining the loss value based on the target optical flow map and the predicted sub-percussion transformation matrix includes: A first loss value is obtained based on the target optical flow map and the predicted optical flow map; The second loss value is obtained through the predicted sub-camera transformation matrix, wherein the second loss value includes at least one of a third loss value for constraining the entire image and a fourth loss value for constraining sparse points, the entire image being obtained through the sub-camera sample image and the predicted sub-camera transformation matrix, and the sparse points being matching points corresponding to each grid point obtained after downsampling the main camera sample image; The loss value is obtained based on the first loss value and the second loss value.

6. The method as described in claim 5, characterized in that, The step of obtaining the second loss value through the predicted secondary transformation matrix includes: The translation amount is obtained based on the predicted sub-perimeter transformation matrix; The compensated predictive sub-photon transformation matrix is ​​obtained by compensating the predicted sub-photon transformation matrix based on the translation amount. The updated sub-camera sample image is obtained by performing an image affine transformation on the sub-camera sample image according to the compensation prediction sub-camera transformation matrix; The difference between the pixel values ​​of each pixel in the aligned sub-camera sample image and the updated sub-camera sample image is calculated, and the third loss value is obtained based on the difference, wherein the aligned sub-camera sample image is an image after the stereo correction algorithm.

7. The method as described in claim 6, characterized in that, The step of obtaining the translation amount based on the predicted sub-perimeter transformation matrix includes: The target optical flow map is downsampled to obtain a downsampled first target optical flow map of the target size; The main camera sample image is divided into multiple image blocks to obtain multiple first grids, and the grid coordinates of each first grid in the multiple first grids are determined, wherein the total number of grids in the multiple first grids is the same as the number of pixels in the downsampled first target optical flow map; The grid optical flow offset information of each first grid is obtained based on the downsampled first target optical flow map; The grid coordinates of any first grid are summed with the grid optical flow offset information of the first grid to obtain the matching point that matches the first grid. This process is repeated to obtain the matching points of all first grids, thus obtaining the set of matching points. Based on the set of matching points and the predicted sub-peripheral transformation matrix, a set of transformation matching points is obtained, and the coordinates of the transformation matching points corresponding to the matching points are obtained respectively. The translation amount is obtained based on the grid coordinates of any first grid and the coordinates of the transformation matching point corresponding to any first grid.

8. The method according to any one of claims 5-7, characterized in that, The step of obtaining the second loss value through the predicted secondary transformation matrix includes: Calculate the first loss value and the second loss value, wherein the first loss value is used to constrain row alignment and the second loss value is used to constrain image translation. The fourth loss value is obtained based on the first loss value and the second loss value.

9. The method as described in claim 8, characterized in that, The calculation of the first loss value and the calculation of the second loss value include: The predicted optical flow map is downsampled to obtain a downsampled second predicted optical flow map of the target size; The main camera sample image is divided into multiple image blocks to obtain multiple second grids, and the grid coordinates of each second grid are determined. The total number of grids in the multiple second grids is the same as the number of pixels in the downsampled second predicted optical flow map. The grid optical flow offset information of each second grid is obtained based on the downsampled second predicted optical flow map; The grid coordinates of any second grid are summed with the grid optical flow offset information of the second grid to obtain the matching point that matches the second grid. This process is repeated to obtain the matching points of all second grids, thereby obtaining the set of sub-image matching points. A transformed sub-photon matching point set is obtained based on the sub-photon matching point set and the predicted sub-photon transformation matrix, wherein each element in the transformed sub-photon matching point set is used to characterize the transformed coordinates of the matching point; The first loss value and the second loss value are calculated based on the grid coordinates of any second grid and the transformed coordinates of the matching point corresponding to any second grid.

10. The method as described in claim 9, characterized in that, The calculation of the first loss value and the second loss value based on the transformed coordinates of the grid coordinates of any second grid and the corresponding matching point of any second grid includes: The first loss value is calculated based on the ordinate value of the grid coordinates and the ordinate value of the transformed coordinates; Solve for the difference between the x-coordinate value of each grid corresponding to all second grids and the x-coordinate value of the corresponding transformed coordinates to obtain multiple differences; The minimum value among the plurality of differences is selected as the second loss value.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it can implement the method described in any one of claims 1-10.

12. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method described in any one of claims 1-10.

13. A computer program product, characterized in that, The computer program product includes a computer program, wherein the computer program, when executed by a processor, can implement the method as described in any one of claims 1-10.

Images