Image correction method and device, electronic equipment and medium

Abstract

The invention relates to an image correction method and device, electronic equipment and a medium. The image correction method comprises the following steps: respectively determining a first feature point set and a second feature point set which are matched in a first image and a second image, wherein a height difference exists between a first camera for collecting the first image and a second camera for collecting the second image in a depth direction; performing first correction processing on the first feature point set and the second feature point set to obtain a third feature point set and a fourth feature point set with aligned rows; according to target feature point pairs in the third feature point set and the fourth feature point set, main points in the internal reference matrix are compensated by taking the condition that the parallax of the target feature point pairs under the preset distance reaches the target parallax as a reference, and target main points are obtained; and performing second correction processing on the first image and the second image according to the internal reference matrix containing the target principal point to obtain a third image and a fourth image with aligned rows, the third image and the fourth image being used for generating a first depth image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image correction method, apparatus, electronic device and medium. Background Technology

[0002] Electronic devices are equipped with multiple cameras. Images captured by two of these cameras can form a depth image, which can be used to improve image quality by determining the depth at different locations within the depth image. However, because there is a certain height difference between the two cameras in the depth direction, the coordinate axes of the depth direction face different orientations in the world coordinate system and the camera coordinate system. This results in inconsistencies in parallax within the same depth plane in the generated depth image. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image correction method, apparatus, electronic device, and medium.

[0004] According to a first aspect of the present disclosure, an image correction method is provided, the image correction method comprising:

[0005] The first feature point set and the second feature point set that match in the first image and the second image are determined respectively. There is a height difference in the depth direction between the first camera that captures the first image and the second camera that captures the second image.

[0006] The first feature point set and the second feature point set are respectively subjected to the first correction process to obtain the row-aligned third feature point set and the fourth feature point set;

[0007] Based on the target feature point pairs in the third feature point set and the fourth feature point set, and taking the disparity of the target feature point pairs at a preset distance as the benchmark to reach the target disparity, the principal points in the intrinsic parameter matrix are compensated to obtain the target principal points.

[0008] Based on the intrinsic parameter matrix containing the target principal point, the first image and the second image are respectively subjected to a second correction process to obtain a row-aligned third image and a fourth image, which are used to generate a first depth image.

[0009] In some embodiments of this disclosure, the step of compensating the principal points in the intrinsic parameter matrix based on the target feature point pairs in the third feature point set and the fourth feature point set, with the disparity of the target feature point pairs at a preset distance reaching the target disparity as a benchmark, to obtain the target principal points, includes:

[0010] Based on the first feature point and the second feature point in the target feature point pair, a third feature point is determined as the second feature point at the preset distance, wherein the first feature point is located in the third feature point set and the second feature point is located in the fourth feature point set;

[0011] Based on the second feature point and the third feature point, the principal points corresponding to the second feature point set are compensated to obtain the target principal point.

[0012] In some embodiments of this disclosure, determining a third feature point as the second feature point at a preset distance based on the first and second feature points in the target feature point pair includes:

[0013] The difference between the x-coordinate of the first feature point and the target disparity is used as the x-coordinate of the third feature point;

[0014] The ordinate of the second feature point is used as the ordinate of the third feature point.

[0015] In some embodiments of this disclosure, the step of compensating the principal points corresponding to the second feature point set based on the second feature point and the third feature point to obtain the target principal point includes:

[0016] The second feature point is transformed to the imaging plane in the world coordinate system to obtain the fourth feature point;

[0017] The third feature point is transformed to a normalized plane in the world coordinate system to obtain the fifth feature point;

[0018] The difference between the abscissa of the fourth feature point and the compensated abscissa is taken as the abscissa of the target principal point. The compensated abscissa is the product of the abscissa of the fifth feature point and the focal length component in the first intrinsic parameter matrix of the second camera.

[0019] The difference between the ordinate of the fourth feature point and the compensated ordinate is taken as the ordinate of the target principal point. The compensated ordinate is the product of the ordinate of the fifth feature point and the focal length component in the first intrinsic parameter matrix of the second camera.

[0020] In some embodiments of this disclosure, before determining the third feature point as the second feature point at the preset distance based on the first and second feature points in the target feature point pair, the step of compensating the principal points in the intrinsic parameter matrix based on the target feature point pairs in the third feature point set and the fourth feature point set, with the disparity of the target feature point pair at the preset distance reaching the target disparity as a benchmark, to obtain the target principal points further includes:

[0021] The pair of feature points whose absolute disparity value is less than a preset disparity in the third and fourth feature point sets is taken as the target feature point pair.

[0022] In some embodiments of this disclosure, the step of performing a first correction process on the first feature point set and the second feature point set respectively to obtain a row-aligned third feature point set and a fourth feature point set includes:

[0023] A loss function is constructed based on the alignment residuals of the first feature point set and the second feature point set in the row direction;

[0024] The loss function is solved using a preset optimization method to obtain multiple first intrinsic parameter matrices and multiple first extrinsic parameter matrices;

[0025] Based on each of the first intrinsic parameter matrices and each of the first extrinsic parameter matrices, the first feature point set and the second feature point set are respectively subjected to the first correction process to obtain the third feature point set and the fourth feature point set.

[0026] In some embodiments of this disclosure, the step of performing a second correction process on the first image and the second image respectively based on an intrinsic parameter matrix containing the target principal point to obtain row-aligned third and fourth images includes:

[0027] The loss function is solved by a preset optimization method to obtain multiple second intrinsic parameter matrices and multiple second extrinsic parameter matrices. The loss function is constructed based on the first feature point set and the second feature point set.

[0028] Based on each of the second intrinsic parameter matrices and each of the second extrinsic parameter matrices, the first image and the second image are respectively subjected to the second correction processing to obtain the third image and the fourth image;

[0029] In the process of solving the intrinsic and extrinsic parameter matrices in the loss function, the x-coordinate of the target principal point contained in the intrinsic parameter matrix remains unchanged.

[0030] In some embodiments of this disclosure, determining the matching first feature point set and second feature point set in the first image and the second image respectively includes:

[0031] Perform viewpoint alignment and brightness consistency processing on the first image and the second image;

[0032] Using a feature matching algorithm, multiple feature point pairs with matching relationships are extracted from the first image and the second image to obtain the fifth feature point set of the first image and the sixth feature point set of the second image;

[0033] The fifth feature point set and the sixth feature point set are normalized respectively to obtain the first feature point set and the second feature point set.

[0034] In some embodiments of this disclosure, the first camera is a periscope telephoto camera, and the second camera is a short-focus camera.

[0035] In some embodiments of this disclosure, after performing a second correction process on the first image and the second image based on an intrinsic parameter matrix containing the target principal point to obtain a third image and a fourth image, the image correction method further includes:

[0036] Based on the third image and the fourth image, generate the first depth image in the camera coordinate system;

[0037] The first depth image is transformed to obtain a second depth image in the world coordinate system.

[0038] In some embodiments of this disclosure, after transforming the first depth image to obtain a second depth image in the world coordinate system, the image correction method further includes:

[0039] Based on the baseline length of the binocular system containing the first and second cameras and the focal length component in the second intrinsic parameter matrix shared by the first and second cameras, the parallax in the second depth image is corrected.

[0040] According to a second aspect of the present disclosure, an image correction apparatus is provided, the image correction apparatus comprising:

[0041] The determining module is configured to determine a first set of feature points and a second set of feature points that match in a first image and a second image, respectively, wherein the first camera that acquires the first image and the second camera that acquires the second image have a height difference in the depth direction;

[0042] A first correction module is configured to perform a first correction process on the first feature point set and the second feature point set respectively to obtain a row-aligned third feature point set and a fourth feature point set.

[0043] The compensation module is configured to compensate the principal points in the intrinsic parameter matrix based on the target feature point pairs in the third feature point set and the fourth feature point set, with the disparity of the target feature point pairs at a preset distance reaching the target disparity as a benchmark, to obtain the target principal points.

[0044] The second correction module is configured to perform a second correction process on the first image and the second image respectively according to the intrinsic parameter matrix containing the target principal point, to obtain a row-aligned third image and a fourth image, the third image and the fourth image being used to generate a first depth image.

[0045] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0046] processor;

[0047] Memory used to store the processor's executable instructions;

[0048] The processor is configured to perform the image correction method described above.

[0049] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to perform the image correction method as described above.

[0050] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0051] First and second feature point sets are determined in the first and second images respectively to obtain two corresponding feature point sets representing the first and second images. Since there is an alignment residual between the first and second feature point sets in the row direction, a first correction process is performed on the first and second feature point sets respectively to obtain row-aligned third and fourth feature point sets. Based on the target feature point pairs in the third and fourth feature point sets, the principal points in the intrinsic parameter matrix are compensated to obtain the target principal points, using the disparity of the target feature point pairs at a preset distance as a benchmark. Based on the intrinsic parameter matrix containing the target principal points, a second correction process is performed on the first and second images respectively to obtain row-aligned third and fourth images used to generate the first depth image. By compensating the principal points in the intrinsic parameter matrix used for the second correction process using the target disparity at a preset distance as a benchmark, the first and second cameras can be moved further apart in the depth direction during the second correction process on the first and second images to reduce the influence of height difference, thereby eliminating the problem of disparity inconsistency in the same depth plane in the depth image.

[0052] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0054] Figure 1-1 This is a schematic diagram of a depth image on a plane of the same depth in a world coordinate system;

[0055] Figure 1-2 This is a schematic diagram of a depth image in the same depth plane under a camera coordinate system;

[0056] Figure 1-3 This is a schematic diagram of a depth image of the same depth plane in another world coordinate system;

[0057] Figure 1-4 This is a schematic diagram of a shooting scene;

[0058] Figure 1-5 This is a schematic diagram of an ideal depth image;

[0059] Figure 1-6 This is a schematic diagram of an actual depth image;

[0060] Figure 2 This is a schematic diagram illustrating an application scenario of an image correction method according to an exemplary embodiment;

[0061] Figure 3 This is a schematic flowchart illustrating an image correction method according to an exemplary embodiment;

[0062] Figure 4 This is a flowchart illustrating an image correction method according to another exemplary embodiment;

[0063] Figure 5 This is a flowchart illustrating an image correction method according to another exemplary embodiment;

[0064] Figure 6 This is a flowchart illustrating an image correction method according to another exemplary embodiment;

[0065] Figure 7 This is a schematic diagram illustrating a feature point transformation process according to an exemplary embodiment;

[0066] Figure 8 This is a flowchart illustrating an image correction method according to another exemplary embodiment;

[0067] Figure 9 This is a flowchart illustrating an image correction method according to another exemplary embodiment;

[0068] Figure 10-1This is a schematic diagram of a depth image in the same depth plane in a camera coordinate system according to an exemplary embodiment;

[0069] Figure 10-2 This is a schematic diagram of a depth image of a plane at the same depth in the world coordinate system, according to another exemplary embodiment.

[0070] Figure 11 This is a block diagram of an image correction apparatus according to an exemplary embodiment;

[0071] Figure 12 This is a block diagram of an electronic device according to an exemplary embodiment.

[0072] In the picture:

[0073] 100 - Determining module; 150 - First calibration module; 200 - Compensation module; 250 - Second calibration module; 400 - Electronic device; 402 - Processing component; 404 - Memory; 406 - Power supply component; 408 - Multimedia component; 410 - Audio component; 412 - Input / output interface; 414 - Sensor component; 416 - Communication component; 420 - Processor. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims. It should also be understood that the term “and / or” as used in this disclosure refers to and includes any or all possible combinations of one or more of the associated listed items.

[0075] With the development of electronic devices, multiple cameras of different types are incorporated into these devices, enabling simultaneous image capture to improve image quality. In some shooting scenarios (such as portrait blurring scenes), two cameras are needed to capture images simultaneously, and a depth image is generated from these two images. By processing the captured images according to the depth at different locations within the depth image (such as depth-of-field blurring), the desired image is generated. However, as... Figures 1-1 to 1-3As shown, due to the height difference between two cameras (such as a periscope telephoto camera and a short-focus camera) in the depth direction (z-axis direction), the orientation of the depth coordinate axis differs between the world coordinate system and the camera coordinate system. This leads to inconsistencies in parallax within the same depth plane in the depth images generated in the camera coordinate system and those converted back to the world coordinate system. In other words, ideally, the parallax of all points in a depth image should be consistent within the same depth plane. However, due to the height difference between the two cameras, the parallax in the depth image gradually changes along one direction. Because of this inconsistency in parallax within the same depth plane, the depth image cannot accurately represent the acquired image, thus affecting the image quality. Figures 1-4 to 1-6 As shown, when there is no height difference between the two cameras, the disparity corresponding to each point on plane 1 in the depth image is the same, and the disparity corresponding to each point on plane 2 in the depth image is also the same. When there is a height difference between the two cameras, the disparity corresponding to each point on plane 1 in the depth image gradually changes along one direction, and the disparity corresponding to each point on plane 2 in the depth image also gradually changes along one direction.

[0076] To address the aforementioned technical issues, this disclosure provides an image correction method. By compensating for the principal points in the intrinsic parameter matrix used for the second correction process, the method pulls the first and second cameras further apart in the depth direction during the second correction process of the first and second images to reduce the influence of the height difference, thereby solving the problem of inconsistent parallax in the same depth plane in the depth image.

[0077] To facilitate understanding, the application scenarios of the image correction method disclosed herein will be described first. For example... Figure 2 As shown, there is a height difference t between the optical center C1 of the first camera and the optical center C2 of the second camera in the depth direction. z There is a horizontal difference t in the horizontal direction. x After the first camera acquires a first image I1 and the second camera acquires a second image I2, a second correction process is performed on the first image I1 and the second image I2 respectively to obtain a third image and a fourth image. Based on the third image and the fourth image, a first depth image D is generated.

[0078] This disclosure provides an image correction method, such as... Figure 3 As shown, the method includes:

[0079] S100. Determine the first feature point set and the second feature point set that match in the first image and the second image respectively. The first camera that acquires the first image and the second camera that acquires the second image have a height difference in the depth direction.

[0080] S200. Perform the first correction process on the first feature point set and the second feature point set respectively to obtain the row-aligned third feature point set and fourth feature point set.

[0081] S300. Based on the target feature point pairs in the third and fourth feature point sets, and taking the disparity of the target feature point pairs at a preset distance as the benchmark to reach the target disparity, the principal points in the intrinsic parameter matrix are compensated to obtain the target principal points.

[0082] S400. Based on the intrinsic parameter matrix containing the target principal point, perform a second correction process on the first image and the second image respectively to obtain a row-aligned third image and a fourth image. The third image and the fourth image are used to generate the first depth image.

[0083] In this embodiment, a first feature point set and a second feature point set matching in the first image and the second image are determined respectively to obtain two feature point sets that correspond to each other and represent the first image and the second image. Since there is an alignment residual between the first feature point set and the second feature point set in the row direction, a first correction process is performed on the first feature point set and the second feature point set respectively to obtain a row-aligned third feature point set and a fourth feature point set. Based on the target feature point pairs in the third feature point set and the fourth feature point set, the principal points in the intrinsic parameter matrix are compensated to obtain the target principal points, with the disparity of the target feature point pairs at a preset distance reaching the target disparity as the benchmark. Based on the intrinsic parameter matrix containing the target principal points, a second correction process is performed on the first image and the second image respectively to obtain a row-aligned third image and a fourth image used to generate the first depth image. By compensating the principal points in the intrinsic parameter matrix used for the second correction process with the target disparity at a preset distance as the benchmark, the first camera and the second camera can be pulled further apart in the depth direction during the second correction process of the first image and the second image to reduce the influence of the height difference, thereby eliminating the problem of disparity inconsistency in the same depth plane in the depth image.

[0084] For example, the preset distance refers to the preset distance between the first or second camera and the preset plane, used to zoom out the first and second cameras. The preset distance can range from 50m to 200m. The preset distance can be 80m, 100m, 120m, etc. Target parallax refers to the expected parallax of a pair of target feature points at the preset distance.

[0085] In one embodiment, the first camera is a periscope telephoto camera, and the second camera is a short-focus camera.

[0086] In this embodiment, because the optical centers of the two cameras in the binocular system consisting of a periscope telephoto camera and a short-focus camera have a height difference in the depth direction, and are used to capture ultra-telephoto portraits, by making the first camera a periscope telephoto camera and the second camera a short-focus camera, the parallax in the same depth plane in the depth image can be made consistent in application scenarios such as portrait blurring, thereby improving the image quality.

[0087] In one embodiment, such as Figure 4 As shown, the first feature point set and the second feature point set matching in the first image and the second image, respectively, in step S100 are determined in the following way:

[0088] S110. Perform viewpoint alignment and brightness consistency processing on the first and second images.

[0089] S120. Using a feature matching algorithm, extract multiple feature point pairs with matching relationships from the first image and the second image to obtain the fifth feature point set of the first image and the sixth feature point set of the second image.

[0090] S130. Normalize the fifth feature point set and the sixth feature point set respectively to obtain the first feature point set and the second feature point set.

[0091] In this embodiment, since the first and second cameras have different field of view and shooting parameters, to avoid feature point matching errors, the first and second images undergo viewpoint alignment and brightness consistency processing. Using a feature matching algorithm, multiple feature point pairs with matching relationships are extracted from the first and second images, resulting in the fifth feature point set of the first image and the sixth feature point set of the second image. Since the intrinsic and extrinsic parameter matrices of the cameras need to be solved during the first and second correction processes, the fifth and sixth feature point sets are normalized respectively to obtain the first and second feature point sets. By extracting and normalizing feature points from the preprocessed first and second images, the accuracy of feature point matching is high, and the intrinsic and extrinsic parameter matrices are easily solved, thereby improving the reliability of image correction.

[0092] For example, the fifth feature point set is denoted as The sixth set of feature points is denoted as Where i represents the matching relationship between feature points, and n represents the number of feature point pairs. That is, under the same i, two feature points match. In the fifth feature point set, each feature point is denoted as . In the sixth set of feature points, each feature point is denoted as...

[0093] For example, in step S130, normalizing the fifth and sixth feature point sets to obtain the first and second feature point sets can be achieved by performing similarity transformations on the fifth and sixth feature point sets respectively. The process of normalizing the fifth feature point set will be described using the fifth feature point set as an example.

[0094] a) Calculate the geometric center

[0095]

[0096] b) Calculate the scaling factor

[0097]

[0098] c) Construct the first similarity transformation matrix S1:

[0099]

[0100] d) Apply the first similarity transformation matrix S1 to the fifth feature point set to obtain the first feature point set.

[0101]

[0102] Using the steps a) to c) above, Replace with Will Replace with The second similarity transformation matrix S2 can be obtained. Applying the second similarity transformation matrix S2 to the sixth feature point set yields the second feature point set.

[0103]

[0104] In the process of normalizing the fifth and sixth feature point sets, it is necessary to translate the point sets so that their centers fall on the origin, and scale the point sets so that the average distance between the coordinates of each point and the center of the circle is...

[0105] In one embodiment, such as Figure 5 As shown, in step S200, the first correction process is performed on the first feature point set and the second feature point set respectively to obtain the row-aligned third feature point set and the fourth feature point set, which are determined in the following way:

[0106] S210. Construct a loss function based on the alignment residuals of the first feature point set and the second feature point set in the row direction.

[0107] S220. Solve the intrinsic and extrinsic parameter matrices in the loss function using a preset optimization method to obtain multiple first intrinsic parameter matrices and multiple first extrinsic parameter matrices.

[0108] S230. Based on each first intrinsic parameter matrix and each first extrinsic parameter matrix, perform the first correction process on the first feature point set and the second feature point set respectively to obtain the third feature point set and the fourth feature point set.

[0109] In this embodiment, since the first and second feature point sets have alignment residuals in the row direction that need to be aligned, a loss function is constructed based on these alignment residuals. Because the intrinsic and extrinsic parameter matrices of the first and second cameras may change at any time, a preset optimization method is used to solve for each intrinsic and extrinsic parameter matrix in the loss function, resulting in multiple first intrinsic and extrinsic parameter matrices. Based on each first intrinsic and extrinsic parameter matrix, a first correction process is performed on the first and second feature point sets respectively, resulting in a third and fourth feature point set. By solving for the first intrinsic and extrinsic parameter matrices to perform the first correction process, the image correction is prevented from being affected by changes in the camera's intrinsic and extrinsic parameter matrices, thereby improving the reliability of image correction.

[0110] For example, in step S210, constructing the loss function based on the alignment residuals of the first and second feature point sets in the row direction can be a modeling method based on the Bouguet method, calculating the alignment residuals of the first and second feature point sets in the row direction to construct the loss function. In step S220, solving for each intrinsic and extrinsic parameter matrix in the loss function using a preset optimization method to obtain multiple first intrinsic and extrinsic parameter matrices can be done using the Levenberg-Marquardt (LM) nonlinear optimization method to solve for each intrinsic and extrinsic parameter matrix in the loss function, obtaining multiple first intrinsic and extrinsic parameter matrices.

[0111] For example, in step S230, the first feature point set and the second feature point set are subjected to a first correction process based on each of the first intrinsic parameter matrices and each of the first extrinsic parameter matrices, respectively, to obtain the third feature point set and the fourth feature point set, which are represented by the following formula:

[0112]

[0113] in, Represents the set of third feature points. Let H1 represent the fourth feature point set, H2 represent the first homography transformation matrix, H2 represent the second homography transformation matrix, K represent the first intrinsic parameter matrix shared by the first and second cameras, K1 represent the first intrinsic parameter matrix of the first camera, K2 represent the first intrinsic parameter matrix of the second camera, R1 represent the first extrinsic parameter matrix of the first camera, and R2 represent the first extrinsic parameter matrix of the second camera.

[0114] In one embodiment, such as Figure 6 As shown, in step S300, based on the target feature point pairs in the third and fourth feature point sets, and taking the disparity of the target feature point pairs at a preset distance reaching the target disparity as a benchmark, the principal points in the intrinsic parameter matrix are compensated, and the target principal points are determined in the following way:

[0115] S310. Based on the first feature point and the second feature point in the target feature point pair, determine the third feature point as the second feature point at a preset distance. The first feature point is located in the set of third feature points, and the second feature point is located in the set of fourth feature points.

[0116] S320. Based on the second feature point and the third feature point, compensate the principal points corresponding to the second feature point set to obtain the target principal point.

[0117] In this embodiment, since it is necessary to ensure that the disparity of the target feature point pair at a preset distance reaches the target disparity, while keeping the first feature point unchanged, a third feature point is determined as the second feature point based on the first and second feature points in the target feature point pair. Based on the second and third feature points, compensation is performed on the principal points corresponding to the second feature point set to obtain the target principal point. By determining the third feature point as the second feature point at a preset distance for principal point compensation, the first and second cameras can be moved further apart in the depth direction during the second correction processing of the first and second images to reduce the influence of the height difference, thereby eliminating the problem of disparity inconsistency in the same depth plane in the depth image.

[0118] For example, the target feature point pair is a pair of feature points selected from a third feature point set and a fourth feature point set. The first feature point in the target feature point pair selected from the third feature point set can be denoted as... The second feature point in the target feature point pair selected from the fourth feature point set can be denoted as...

[0119] In one embodiment, the determination of the third feature point as the second feature point at a preset distance in step S310, based on the first and second feature points in the target feature point pair, is made in the following manner:

[0120] The difference between the x-coordinate of the first feature point and the target disparity is used as the x-coordinate of the third feature point.

[0121] Use the ordinate of the second feature point as the ordinate of the third feature point.

[0122] In this embodiment, since the first and second cameras are arranged horizontally, the parallax is reflected by the horizontal coordinate, and the vertical difference between the optical centers is 0. The difference between the horizontal coordinate of the first feature point and the target parallax is used as the horizontal coordinate of the third feature point, so that the difference between the horizontal coordinates of the first and third feature points is the target parallax, satisfying the requirement that the parallax between the first and third feature points at a preset distance is the target parallax. The vertical coordinate of the second feature point is used as the vertical coordinate of the third feature point, keeping the vertical coordinate of the third feature point unchanged. By setting the horizontal coordinate of the third feature point, the parallax of the target feature point at the preset distance reaches the target parallax, which can reduce the influence of the height difference by moving the first and second cameras further away in the depth direction, thereby eliminating the problem of inconsistency in parallax under the same depth plane in the depth image.

[0123] For example, the first feature point in the target feature point pair at a preset distance can be denoted as... The third feature point can be denoted as... FarDisp represents target parallax.

[0124] In one embodiment, in step S320, the principal points corresponding to the second feature point set are compensated based on the second feature points and the third feature points to obtain the target principal points, which are determined in the following way:

[0125] The second feature point is transformed to the imaging plane in the world coordinate system to obtain the fourth feature point.

[0126] The third feature point is transformed to the normalized plane in the world coordinate system to obtain the fifth feature point.

[0127] The difference between the x-coordinate of the fourth feature point and the compensated x-coordinate is taken as the x-coordinate of the target principal point. The compensated x-coordinate is the product of the x-coordinate of the fifth feature point and the focal length component in the first intrinsic parameter matrix of the second camera.

[0128] The difference between the ordinate of the fourth feature point and the compensated ordinate is taken as the ordinate of the target principal point. The compensated ordinate is the product of the ordinate of the fifth feature point and the focal length component in the first intrinsic parameter matrix of the second camera.

[0129] In this embodiment, since the second and third feature points are feature points in the camera coordinate system, they need to be transformed to the world coordinate system for principal point compensation. The second feature point is transformed to the imaging plane in the world coordinate system, and the feature point is transformed using the inverse of the second homography transformation matrix to obtain the fourth feature point. The third feature point is transformed to the normalized plane in the world coordinate system, and the feature point is transformed using the inverse of the second homography transformation matrix excluding the first intrinsic parameter matrix of the second camera to obtain the fifth feature point. The difference between the abscissa of the fourth feature point and the compensated abscissa is used as the abscissa of the target principal point, thus obtaining the abscissa of the target principal point by compensating for the abscissa of the fourth feature point. The difference between the ordinate of the fourth feature point and the compensated ordinate is used as the ordinate of the target principal point, thus obtaining the ordinate of the target principal point by compensating for the ordinate of the fourth feature point. By transforming the second and third feature points to the world coordinate system, the coordinate axes in the deflected depth direction are restored before principal point compensation, thereby improving the reliability of image correction.

[0130] For example, the fourth feature point can be denoted as The fifth feature point can be denoted as The target principal point can be denoted as The transformation of the second feature point to the imaging plane in the world coordinate system in the above steps yields the fourth feature point, which is represented by the following formula:

[0131]

[0132] The third feature point, transformed to the normalized plane in the world coordinate system in the above steps, is represented by the fifth feature point using the following formula:

[0133]

[0134] The difference between the x-coordinate of the fourth feature point and the compensated x-coordinate, as used as the x-coordinate of the target principal point in the above steps, is expressed by the following formula:

[0135]

[0136] Among them, f K2 This represents the focal length component in the first intrinsic parameter matrix of the second camera.

[0137] The difference between the ordinate of the fourth feature point and the compensated ordinate, as used as the ordinate of the target principal point in the above steps, is expressed by the following formula:

[0138]

[0139] For example, such as Figure 7 As shown, the fourth feature point Point 0 is a black point located on the image plane in the world coordinate system. If the principal point (C) is not defined... x C y Compensation, for the fourth feature point After transformation to the normalized plane, the sixth feature point is obtained. That is, black point 1. The sixth feature point... After transforming to the camera coordinate system, the second feature point is obtained. That is, black point 2. If we perform principal point (C) x C y Compensation yields the target principal point The fourth feature point After transformation to the normalized plane, the fifth feature point is obtained. That is, gray point 1. The fifth feature point... After transforming to the camera coordinate system, the third feature point is obtained. That is, gray point 2. It can be seen that when performing the principal point (C... x C y After compensation, the first and second cameras can be moved further apart in the depth direction to reduce the impact of the height difference, thereby eliminating the problem of inconsistent parallax in the same depth plane in the depth image.

[0140] It is understandable that in step S300, based on the target feature point pairs in the third and fourth feature point sets, and using the disparity of the target feature point pairs at a preset distance reaching the target disparity as a benchmark, compensation is performed on the principal points in the intrinsic parameter matrix to obtain the target principal points. This can be achieved by determining and compensating the principal points corresponding to the second feature point set based on the third feature point (which is the second feature point), or by determining and compensating the principal points corresponding to the first feature point set based on the third feature point (which is the first feature point). The method of determining and compensating the principal points corresponding to the first feature point set based on the third feature point (which is the first feature point) to obtain the target principal points is similar to the method of determining and compensating the principal points corresponding to the second feature point set based on the third feature point (which is the second feature point), and will not be elaborated upon here.

[0141] In one embodiment, before determining the third feature point as the second feature point at a preset distance based on the first and second feature points in the target feature point pair in step S310, step S300, which involves compensating the principal points in the intrinsic parameter matrix based on the target feature point pairs in the third and fourth feature point sets, with the disparity of the target feature point pairs at the preset distance reaching the target disparity as a benchmark, to obtain the target principal points, further includes:

[0142] The pair of feature points whose absolute disparity value is less than the preset disparity in the third and fourth feature point sets are selected as the target feature point pair.

[0143] In this embodiment, since there are multiple feature point pairs in the third and fourth feature point sets, and the absolute disparity values ​​of these multiple feature point pairs are different, the absolute disparity value reflects the distance in the corrected camera coordinate system. By taking a feature point pair in the third and fourth feature point sets whose absolute disparity value is less than a preset disparity as the target feature point pair, the distance in the corrected camera coordinate system is close to the farthest distance to reduce the influence of the height difference, thereby eliminating the problem of disparity inconsistency in the same depth plane in the depth image.

[0144] For example, in the above steps, selecting a feature point pair in the third and fourth feature point sets whose absolute disparity value is less than a preset disparity as the target feature point pair can be achieved by sorting the feature point pairs in the third and fourth feature point sets in ascending order of absolute disparity value. Alternatively, the feature point pair with the smallest absolute disparity value can be selected as the target feature point pair, or a feature point pair with a preset number can be selected as the target feature point pair. The feature point pair with the preset number is close to, but not the same as, the feature point pair with the smallest absolute disparity value, thus avoiding errors in feature point pair extraction and transformation that could cause the actual absolute disparity value of the target feature point pair to be greater than or equal to the preset disparity.

[0145] In one embodiment, such as Figure 8 As shown, in step S400, the first and second images are subjected to a second correction process based on the intrinsic parameter matrix containing the target principal point, and the row-aligned third and fourth images are determined in the following way:

[0146] S410. Solve the intrinsic and extrinsic parameter matrices in the loss function using a preset optimization method to obtain multiple second intrinsic and extrinsic parameter matrices. The loss function is constructed based on the first feature point set and the second feature point set.

[0147] S420. Based on each second intrinsic parameter matrix and each second extrinsic parameter matrix, perform second correction processing on the first image and the second image respectively to obtain the third image and the fourth image.

[0148] In the process of solving for each intrinsic and extrinsic parameter matrix in the loss function, the x-coordinate of the target principal point contained in the intrinsic parameter matrix remains unchanged.

[0149] In this embodiment, the x-coordinate of the principal point determines the disparity range, while the y-coordinate determines the row alignment residual. The x-coordinate of the target principal point improves depth, while the y-coordinate increases row alignment error, necessitating a re-solution of the intrinsic and extrinsic parameter matrices. Using a preset optimization method, while keeping the x-coordinate of the target principal point in the intrinsic parameter matrix unchanged, multiple second intrinsic and extrinsic parameter matrices are obtained by solving the loss function. Based on these second intrinsic and extrinsic parameter matrices, the first and second images undergo a second correction process to obtain the third and fourth images. By keeping the x-coordinate of the target principal point in the intrinsic parameter matrix unchanged and solving the second intrinsic and extrinsic parameter matrices for the second correction process, the problem of disparity inconsistency in the same depth plane in the depth image is eliminated, preventing the first and second cameras, which are pulled away in the depth direction, from being pulled back.

[0150] For example, when compensating for the principal points corresponding to the first set of feature points, the target principal point is located in the intrinsic parameter matrix of the first camera. When compensating for the principal points corresponding to the second set of feature points, the target principal point is located in the intrinsic parameter matrix of the second camera.

[0151] For example, the method in step S410 of solving for each intrinsic parameter matrix and each extrinsic parameter matrix in the loss function using a preset optimization method to obtain multiple second intrinsic parameter matrices and multiple second extrinsic parameter matrices can be the same as the method in step S220 of solving for each intrinsic parameter matrix and each extrinsic parameter matrix in the loss function using a preset optimization method to obtain multiple first intrinsic parameter matrices and multiple first extrinsic parameter matrices, and will not be elaborated here. In step S420, based on each second intrinsic parameter matrix and each second extrinsic parameter matrix, the first image and the second image are respectively subjected to a second correction process to obtain a third image and a fourth image. This can be achieved by transforming the first image and the second image using the adjusted first homography transformation matrix and the second homography transformation matrix, respectively. Here, the intrinsic parameter matrix in the adjusted first homography transformation matrix and the second homography transformation matrix is ​​the second intrinsic parameter matrix, and the extrinsic parameter matrix is ​​the second extrinsic parameter matrix.

[0152] For example, in step S420, the first image and the second image are subjected to a second correction process based on each of the second intrinsic parameter matrices and each of the second extrinsic parameter matrices, and the third image and the fourth image are determined by the following formula:

[0153]

[0154] in, This represents the third image. I1 represents the fourth image, I2 represents the first image, and I2 represents the second image. This represents the adjusted first homography transformation matrix. K represents the adjusted second homography transformation matrix. * This represents the second intrinsic parameter matrix shared by the first and second cameras. This represents the second intrinsic parameter matrix of the first camera. This represents the second intrinsic parameter matrix of the second camera. This represents the second extrinsic parameter matrix of the first camera. This represents the second extrinsic parameter matrix of the second camera.

[0155] In one embodiment, after performing a second correction process on the first and second images based on the intrinsic parameter matrix containing the target principal point in step S400 to obtain the third and fourth images, the image correction method further includes:

[0156] Based on the third and fourth images, generate the first depth image in the camera coordinate system.

[0157] The first depth image is transformed to obtain the second depth image in the world coordinate system.

[0158] In this embodiment, after obtaining the third and fourth images that have undergone the second correction process, a first depth image in the camera coordinate system is generated based on the third and fourth images. Since the depth of the depth image needs to be combined with the depth of the scene in the world coordinate system for application, the first depth image is transformed to obtain a second depth image in the world coordinate system. By generating the first depth image and converting it to the second depth image, image processing can be performed in conjunction with the depth of the second depth image in the corresponding scene, thereby improving the reliability of image processing.

[0159] For example, the transformation of the first depth image in the above steps to obtain the second depth image in the world coordinate system is determined by the following formula:

[0160]

[0161] Where, x b The x-coordinate of a point in the first depth image is represented by y. b The x-coordinate represents the ordinate of that point in the first depth image. w The x-coordinate of that point in the second depth image is represented by y. w This represents the ordinate of the point in the second depth image.

[0162] In one embodiment, after transforming the first depth image in the above steps to obtain the second depth image in the world coordinate system, the image correction method further includes:

[0163] Based on the baseline length of the binocular system containing the first and second cameras, and the focal length component in the second intrinsic parameter matrix shared by the first and second cameras, the parallax in the second depth image is corrected.

[0164] In this embodiment, due to the height difference between the first camera and the second camera in the depth direction, the parallax in the second depth image needs to be corrected. Based on the baseline length of the binocular system containing the first and second cameras and the focal length component in the shared second intrinsic parameter matrix of the first and second cameras, the parallax in the second depth image is corrected, ensuring that the second depth image accurately represents the first image. By correcting the parallax in the second depth image, changes in parallax due to the height difference are avoided, thereby improving the reliability of image correction.

[0165] For example, in the above steps, the disparity correction in the second depth image is determined by the following formula based on the baseline length of the binocular system containing the first and second cameras and the focal length component in the second intrinsic parameter matrix shared by the first and second cameras:

[0166]

[0167] Where Baseline represents the baseline length, f represents the focal length component in the second intrinsic parameter matrix shared by the first and second cameras, and d b D represents the parallax in the camera coordinate system. b d represents the depth in the camera coordinate system. w D represents the parallax in the world coordinate system. w X represents depth in the world coordinate system. w and Y w These are parameters that become invalid after the transformation. The baseline length can be expressed as... t x t represents the horizontal difference in the horizontal direction between the optical centers of the first and second cameras. y t represents the vertical difference in the vertical direction between the optical centers of the first and second cameras. z Indicates the height difference.

[0168] This disclosure provides an image correction method, such as... Figure 9 The method includes:

[0169] S500: Perform viewpoint alignment and brightness consistency processing on the first and second images.

[0170] S510. Using a feature matching algorithm, extract multiple feature point pairs with matching relationships from the first image and the second image to obtain the fifth feature point set and the sixth feature point set.

[0171] S520. Normalize the fifth feature point set and the sixth feature point set respectively to obtain the first feature point set and the second feature point set.

[0172] S530. Construct a loss function based on the alignment residuals of the first feature point set and the second feature point set in the row direction.

[0173] S540. Solve the intrinsic and extrinsic parameter matrices in the loss function using a nonlinear optimization method to obtain multiple first intrinsic parameter matrices and multiple first extrinsic parameter matrices.

[0174] S550. Based on each first intrinsic parameter matrix and each first extrinsic parameter matrix, perform the first correction process on the first feature point set and the second feature point set respectively to obtain the third feature point set and the fourth feature point set.

[0175] S560. Select a pair of feature points in the third and fourth feature point sets whose absolute disparity is less than a preset disparity as the target feature point pair.

[0176] S570. The difference between the x-coordinate of the first feature point in the target feature point pair and the target disparity is used as the x-coordinate of the third feature point.

[0177] S580. Use the ordinate of the second feature point in the target feature point pair as the ordinate of the third feature point.

[0178] S590. Transform the second feature point to the imaging plane in the world coordinate system to obtain the fourth feature point.

[0179] S600. Transform the third feature point to the normalized plane in the world coordinate system to obtain the fifth feature point.

[0180] S610. The difference between the x-coordinate of the fourth feature point and the compensated x-coordinate is taken as the x-coordinate of the target principal point.

[0181] S620. The difference between the ordinate of the fourth feature point and the compensated ordinate is taken as the ordinate of the target principal point.

[0182] S630. Using a nonlinear optimization method, while keeping the x-coordinate of the target principal point in the intrinsic parameter matrix of the second camera unchanged, solve for each intrinsic parameter matrix and each extrinsic parameter matrix in the loss function to obtain multiple second intrinsic parameter matrices and multiple second extrinsic parameter matrices.

[0183] S640. Based on each of the second intrinsic parameter matrices and each of the second extrinsic parameter matrices, perform second correction processing on the first image and the second image respectively to obtain the third image and the fourth image.

[0184] S650. Based on the third and fourth images, generate the first depth image in the camera coordinate system.

[0185] S660. Transform the first depth image to obtain the second depth image in the world coordinate system.

[0186] S670. Based on the baseline length and the focal length component in the second intrinsic parameter matrix shared by the first and second cameras, the parallax in the second depth image is corrected.

[0187] In this embodiment, the first and second images are subjected to viewpoint alignment and brightness consistency processing to ensure that the viewpoints of the first and second images are aligned and their brightness is consistent, thus avoiding interference with feature point extraction. Using a feature matching algorithm, multiple feature point pairs with matching relationships are extracted from the first and second images, resulting in a fifth and a sixth feature point set. The fifth and sixth feature point sets are then normalized to obtain a first and a second feature point set, balancing the loss descent rate of the intrinsic and extrinsic parameter matrices in the optimization loss function. Based on the alignment residuals of the first and second feature point sets in the row direction, a loss function is constructed to determine the intrinsic and extrinsic parameter matrices used for row alignment. The intrinsic and extrinsic parameter matrices in the loss function are solved using a nonlinear optimization method to obtain multiple first intrinsic and extrinsic parameter matrices, which are used for preliminary correction. Based on each first intrinsic and extrinsic parameter matrix, the first and second feature point sets are subjected to a first correction process, resulting in a third and a fourth feature point set. The first feature point is selected from the third and fourth feature point sets, where the absolute disparity is less than a preset disparity. This is used as the target feature point pair. The difference between the x-coordinate of the first feature point in the target feature point pair and the target disparity is used as the x-coordinate of the third feature point. The y-coordinate of the second feature point in the target feature point pair is also used as the y-coordinate of the third feature point. The second feature point is then transformed to the imaging plane in the world coordinate system, and the inverse of the second homography transformation matrix is ​​used to transform the feature point, resulting in the fourth feature point. The third feature point is then transformed to the normalized plane in the world coordinate system, and the inverse of the second homography transformation matrix (excluding the first intrinsic parameter matrix of the second camera) is used to transform the feature point, resulting in the fifth feature point. The difference between the x-coordinate and the compensated x-coordinate of the fourth feature point is used as the x-coordinate of the target principal point, and the difference between the y-coordinate and the compensated y-coordinate of the fourth feature point is used as the y-coordinate of the target principal point, thus obtaining the target principal point. Using a nonlinear optimization method, while keeping the x-coordinate of the target principal point in the intrinsic parameter matrix of the second camera unchanged, multiple second intrinsic parameter matrices and multiple second extrinsic parameter matrices are obtained by solving the intrinsic and extrinsic parameter matrices in the loss function. Based on these second intrinsic and extrinsic parameter matrices, the first and second images are subjected to a second correction process to obtain a third and a fourth image. After obtaining the second-corrected third and fourth images, a first depth image in the camera coordinate system is generated based on them. Since the depth of the depth image needs to be combined with the depth data in the world coordinate system for scene application, the first depth image is transformed to obtain a second depth image in the world coordinate system. Based on the baseline length and the focal length component in the second intrinsic parameter matrix shared by the first and second cameras, the disparity in the second depth image is corrected to eliminate the influence of height differences.By using the target disparity at a preset distance as a benchmark to compensate for the principal points in the intrinsic parameter matrix used for the second correction processing, the first camera and the second camera can be pulled further apart in the depth direction during the second correction processing of the first image and the second image to reduce the influence of the height difference, thereby eliminating the problem of disparity inconsistency in the same depth plane in the depth image.

[0188] For example, such as Figure 10-1 and Figure 10-2 As shown, after adopting the image correction method provided in this disclosure, the parallax in the depth image generated in the camera coordinate system and converted back to the world coordinate system is consistent with that in the depth plane.

[0189] In one exemplary embodiment, an image correction apparatus is provided for implementing the method described above. (Reference) Figure 11 As shown, the image correction device may include a determining module 100, a first correction module 150, a compensation module 200, and a second correction module 250. During the implementation of the above method,

[0190] The determination module 100 is configured to determine the first feature point set and the second feature point set that match in the first image and the second image, respectively, wherein the first camera that acquires the first image and the second camera that acquires the second image have a height difference in the depth direction.

[0191] The first correction module 150 is configured to perform a first correction process on the first feature point set and the second feature point set respectively to obtain a row-aligned third feature point set and a fourth feature point set.

[0192] The compensation module 200 is configured to compensate the principal points in the intrinsic parameter matrix based on the target feature point pairs in the third feature point set and the fourth feature point set, with the disparity of the target feature point pairs at a preset distance reaching the target disparity as a benchmark, to obtain the target principal points.

[0193] The second correction module 250 is configured to perform a second correction process on the first image and the second image respectively according to the intrinsic parameter matrix containing the target principal point, so as to obtain a row-aligned third image and a fourth image, which are used to generate a first depth image.

[0194] In one exemplary embodiment, an image correction apparatus is provided, wherein a determining module 100 is configured to:

[0195] The first and second images are processed for viewpoint alignment and brightness consistency.

[0196] Using a feature matching algorithm, multiple feature point pairs with matching relationships are extracted from the first image and the second image to obtain the fifth feature point set of the first image and the sixth feature point set of the second image.

[0197] Normalize the fifth and sixth feature point sets respectively to obtain the first and second feature point sets.

[0198] In one exemplary embodiment, an image correction apparatus is provided, wherein a first correction module 150 is configured to:

[0199] A loss function is constructed based on the alignment residuals of the first and second feature point sets in the row direction.

[0200] The intrinsic and extrinsic parameter matrices in the loss function are solved using a preset optimization method to obtain multiple first intrinsic parameter matrices and multiple first extrinsic parameter matrices.

[0201] Based on each of the first intrinsic parameter matrices and each of the first extrinsic parameter matrices, the first feature point set and the second feature point set are respectively subjected to the first correction process to obtain the third feature point set and the fourth feature point set.

[0202] In one exemplary embodiment, an image correction apparatus is provided, wherein a compensation module 200 is configured to:

[0203] Based on the first and second feature points in the target feature point pair, a third feature point is determined as the second feature point at a preset distance. The first feature point is located in the set of third feature points, and the second feature point is located in the set of fourth feature points.

[0204] Based on the second and third feature points, the principal points corresponding to the second feature point set are compensated to obtain the target principal points.

[0205] In one exemplary embodiment, an image correction apparatus is provided, wherein a compensation module 200 is configured to:

[0206] The difference between the x-coordinate of the first feature point and the target disparity is used as the x-coordinate of the third feature point.

[0207] Use the ordinate of the second feature point as the ordinate of the third feature point.

[0208] In one exemplary embodiment, an image correction apparatus is provided, wherein a compensation module 200 is configured to:

[0209] The second feature point is transformed to the imaging plane in the world coordinate system to obtain the fourth feature point.

[0210] The third feature point is transformed to the normalized plane in the world coordinate system to obtain the fifth feature point.

[0211] The difference between the x-coordinate of the fourth feature point and the compensated x-coordinate is taken as the x-coordinate of the target principal point. The compensated x-coordinate is the product of the x-coordinate of the fifth feature point and the focal length component in the first intrinsic parameter matrix of the second camera.

[0212] The difference between the ordinate of the fourth feature point and the compensated ordinate is taken as the ordinate of the target principal point. The compensated ordinate is the product of the ordinate of the fifth feature point and the focal length component in the first intrinsic parameter matrix of the second camera.

[0213] In one exemplary embodiment, an image correction apparatus is provided, wherein a compensation module 200 is configured to:

[0214] The pair of feature points whose absolute disparity value is less than the preset disparity in the third and fourth feature point sets are selected as the target feature point pair.

[0215] In one exemplary embodiment, an image correction apparatus is provided, wherein a second correction module 250 is configured to:

[0216] The loss function is solved by a preset optimization method to obtain multiple second intrinsic parameter matrices and multiple second extrinsic parameter matrices. The loss function is constructed based on the first feature point set and the second feature point set.

[0217] Based on each of the second intrinsic parameter matrices and the second extrinsic parameter matrices, the first image and the second image are subjected to the second correction process respectively to obtain the third image and the fourth image.

[0218] In the process of solving for each intrinsic and extrinsic parameter matrix in the loss function, the x-coordinate of the target principal point contained in the intrinsic parameter matrix remains unchanged.

[0219] In one exemplary embodiment, an image correction apparatus is provided, the apparatus further comprising:

[0220] The generation module is configured to generate a first depth image in the camera coordinate system based on the third and fourth images.

[0221] The first depth image is transformed to obtain the second depth image in the world coordinate system.

[0222] In one exemplary embodiment, an image correction apparatus is provided, wherein a second correction module 250 is configured to:

[0223] Based on the baseline length of the binocular system containing the first and second cameras, and the focal length component in the second intrinsic parameter matrix shared by the first and second cameras, the parallax in the second depth image is corrected.

[0224] In one exemplary embodiment, an electronic device is provided, such as a mobile phone, a laptop computer, a tablet computer, and a wearable device.

[0225] refer to Figure 12 As shown, the electronic device 400 may include one or more of the following components: processing component 402, memory 404, power supply component 406, multimedia component 408, audio component 410, input / output (I / O) interface 412, sensor component 414, and communication component 416.

[0226] Processing component 402 typically controls the overall operation of electronic device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0227] Memory 404 is configured to store various types of data to support the operation of electronic device 400. Examples of this data include instructions for any application or method operating on electronic device 400, contact data, phonebook data, messages, pictures, videos, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0228] Power supply component 406 provides power to various components of electronic device 400. Power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 400.

[0229] Multimedia component 408 includes a screen that provides an output interface between electronic device 400 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera module and / or a rear-facing camera module. When electronic device 400 is in an operating mode, such as shooting mode or video mode, the front-facing camera module and / or rear-facing camera module may receive external multimedia data. Each front-facing camera module and rear-facing camera module may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0230] Audio component 410 is configured to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) configured to receive external audio signals when electronic device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0231] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0232] Sensor assembly 414 includes one or more sensors for providing state assessments of various aspects of electronic device 400. For example, sensor assembly 414 may detect the on / off state of electronic device 400, the relative positioning of components such as the display and keypad of electronic device 400, changes in position of electronic device 400 or a component of electronic device 400, the presence or absence of user contact with electronic device 400, orientation or acceleration / deceleration of electronic device 400, and temperature changes of electronic device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0233] Communication component 416 is configured to facilitate wired or wireless communication between electronic device 400 and other terminals. Electronic device 400 can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0234] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing terminals (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods shown in the above embodiments or combinations thereof.

[0235] In one exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, which can be executed by a processor 420 of an electronic device 400 to perform the methods shown in the embodiments or combinations thereof. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage terminal, etc. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the methods shown in the embodiments or combinations thereof.

[0236] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0237] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0238] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0239] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0240] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image correction method, characterized in that, The image correction method includes: The first feature point set and the second feature point set that match in the first image and the second image are determined respectively. There is a height difference in the depth direction between the first camera that captures the first image and the second camera that captures the second image. The first feature point set and the second feature point set are respectively subjected to the first correction process to obtain the row-aligned third feature point set and the fourth feature point set; Based on the target feature point pairs in the third feature point set and the fourth feature point set, and taking the disparity of the target feature point pairs at a preset distance as the benchmark to reach the target disparity, the principal points in the intrinsic parameter matrix are compensated to obtain the target principal points. Based on the intrinsic parameter matrix containing the target principal point, the first image and the second image are respectively subjected to a second correction process to obtain a row-aligned third image and a fourth image, which are used to generate a first depth image.

2. The image correction method according to claim 1, characterized in that, The step of compensating the principal points in the intrinsic parameter matrix based on the target feature point pairs in the third and fourth feature point sets, using the disparity of the target feature point pairs at a preset distance reaching the target disparity as a benchmark, to obtain the target principal points includes: Based on the first feature point and the second feature point in the target feature point pair, a third feature point is determined as the second feature point at the preset distance, wherein the first feature point is located in the third feature point set and the second feature point is located in the fourth feature point set; Based on the second feature point and the third feature point, the principal points corresponding to the second feature point set are compensated to obtain the target principal point.

3. The image correction method according to claim 2, characterized in that, The step of determining a third feature point as the second feature point at a preset distance based on the first and second feature points in the target feature point pair includes: The difference between the x-coordinate of the first feature point and the target disparity is used as the x-coordinate of the third feature point; The ordinate of the second feature point is used as the ordinate of the third feature point.

4. The image correction method according to claim 2, characterized in that, The step of compensating the principal points corresponding to the second feature point set based on the second feature point and the third feature point to obtain the target principal point includes: The second feature point is transformed to the imaging plane in the world coordinate system to obtain the fourth feature point; The third feature point is transformed to a normalized plane in the world coordinate system to obtain the fifth feature point; The difference between the abscissa of the fourth feature point and the compensated abscissa is taken as the abscissa of the target principal point. The compensated abscissa is the product of the abscissa of the fifth feature point and the focal length component in the first intrinsic parameter matrix of the second camera. The difference between the ordinate of the fourth feature point and the compensated ordinate is taken as the ordinate of the target principal point. The compensated ordinate is the product of the ordinate of the fifth feature point and the focal length component in the first intrinsic parameter matrix of the second camera.

5. The image correction method according to claim 2, characterized in that, Before determining the third feature point as the second feature point at the preset distance based on the first and second feature points in the target feature point pair, the step of compensating the principal points in the intrinsic parameter matrix based on the target feature point pairs in the third feature point set and the fourth feature point set, with the disparity of the target feature point pair at the preset distance reaching the target disparity as a benchmark, to obtain the target principal point further includes: The pair of feature points whose absolute disparity value is less than a preset disparity in the third and fourth feature point sets is taken as the target feature point pair.

6. The image correction method according to claim 1, characterized in that, The step of performing a first correction process on the first feature point set and the second feature point set respectively to obtain a row-aligned third feature point set and a fourth feature point set includes: A loss function is constructed based on the alignment residuals of the first feature point set and the second feature point set in the row direction; The loss function is solved using a preset optimization method to obtain multiple first intrinsic parameter matrices and multiple first extrinsic parameter matrices; Based on each of the first intrinsic parameter matrices and each of the first extrinsic parameter matrices, the first feature point set and the second feature point set are respectively subjected to the first correction process to obtain the third feature point set and the fourth feature point set.

7. The image correction method according to claim 1, characterized in that, The step of performing a second correction process on the first image and the second image respectively based on the intrinsic parameter matrix containing the target principal point to obtain row-aligned third and fourth images includes: The loss function is solved by a preset optimization method to obtain multiple second intrinsic parameter matrices and multiple second extrinsic parameter matrices. The loss function is constructed based on the first feature point set and the second feature point set. Based on each of the second intrinsic parameter matrices and each of the second extrinsic parameter matrices, the first image and the second image are respectively subjected to the second correction processing to obtain the third image and the fourth image; In the process of solving the intrinsic and extrinsic parameter matrices in the loss function, the x-coordinate of the target principal point contained in the intrinsic parameter matrix remains unchanged.

8. The image correction method according to claim 1, characterized in that, The step of determining the matching first feature point set and second feature point set in the first image and the second image respectively includes: Perform viewpoint alignment and brightness consistency processing on the first image and the second image; Using a feature matching algorithm, multiple feature point pairs with matching relationships are extracted from the first image and the second image to obtain the fifth feature point set of the first image and the sixth feature point set of the second image; The fifth feature point set and the sixth feature point set are normalized respectively to obtain the first feature point set and the second feature point set.

9. The image correction method according to claim 1, characterized in that, The first camera is a periscope telephoto camera, and the second camera is a short-focus camera.

10. The image correction method according to any one of claims 1 to 9, characterized in that, After performing a second correction process on the first image and the second image based on the intrinsic parameter matrix containing the target principal point to obtain the third image and the fourth image, the image correction method further includes: Based on the third image and the fourth image, generate the first depth image in the camera coordinate system; The first depth image is transformed to obtain a second depth image in the world coordinate system.

11. The image correction method according to claim 10, characterized in that, After transforming the first depth image to obtain a second depth image in the world coordinate system, the image correction method further includes: Based on the baseline length of the binocular system containing the first and second cameras and the focal length component in the second intrinsic parameter matrix shared by the first and second cameras, the parallax in the second depth image is corrected.

12. An image correction device, characterized in that, The image correction device includes: The determining module is configured to determine a first set of feature points and a second set of feature points that match in a first image and a second image, respectively, wherein the first camera that acquires the first image and the second camera that acquires the second image have a height difference in the depth direction; A first correction module is configured to perform a first correction process on the first feature point set and the second feature point set respectively to obtain a row-aligned third feature point set and a fourth feature point set. The compensation module is configured to compensate the principal points in the intrinsic parameter matrix based on the target feature point pairs in the third feature point set and the fourth feature point set, with the disparity of the target feature point pairs at a preset distance reaching the target disparity as a benchmark, to obtain the target principal points. The second correction module is configured to perform a second correction process on the first image and the second image respectively according to the intrinsic parameter matrix containing the target principal point, to obtain a row-aligned third image and a fourth image, the third image and the fourth image being used to generate a first depth image.

13. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to perform the image correction method as described in any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the image correction method as described in any one of claims 1 to 11.

Images