Deep computation method, apparatus, computer device and medium

Abstract

The present disclosure relates to a depth calculation method and device, a computer device and a medium, comprising: calculating an initial relative displacement between a main image and a sub-image in a to-be-processed image; moving the sub-image according to the initial relative displacement, so that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement; determining a main camera corner point in the main image, and determining a corresponding sub-camera corner point of each main camera corner point in the sub-image to obtain a corner point pair set; determining a mis-matched corner point pair in the corner point pair set, and deleting the mis-matched corner point pair from the main image and the sub-image to obtain an effective main image and an effective sub-image; moving the effective sub-image back to the initial position in reverse, and performing affine transformation on the sub-camera corner points in the reversed and moved effective sub-image to the effective main image; and determining the depth of a target object according to the sub-camera corner points in the effective main image after affine transformation and the main camera corner points in the effective main image. The accuracy of determining the depth of the target object can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to depth calculation methods, apparatus, computer devices, and media. Background Technology

[0002] Based on the principle of perceiving distance by the parallax of the two eyes when viewing the same object, the distance between the target object and the shooting source can be determined by the parallax of the images captured by the left and right cameras of a binocular camera. Currently, for an image of the target object, the depth value of each feature point in the image is determined based on the parallax of the left and right cameras. Then, based on the depth value of each feature point, the distance between the target object and the shooting source is determined, and thus the spatial position of the target object is determined.

[0003] In related technologies, for images of a target object acquired by left and right cameras, the imaging point of the target object on one camera and the corresponding imaging point on the other camera are determined. Based on the homography matrix, the depth of the target object is determined through the two imaging points, thereby determining the spatial position of the target object. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a deep computing method, apparatus, computer device, and medium.

[0005] According to a first aspect of the present disclosure, a depth calculation method is provided, comprising:

[0006] Calculate the initial relative displacement between the main image and the secondary image in the image to be processed. The image to be processed is an image obtained by the stereo camera based on the target object. The main image is an image captured by the main camera of the stereo camera, and the secondary image is an image captured by the secondary camera of the stereo camera.

[0007] The sub-image is moved according to the initial relative displacement, such that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement.

[0008] Determine the main camera corner points in the main image, and determine the corresponding secondary camera corner points for each main camera corner point in the secondary image to obtain a set of corner point pairs;

[0009] Identify the mismatched corner pairs in the set of corner pairs, and delete the mismatched corner pairs from the main image and the sub-image to obtain a valid main image and a valid sub-image;

[0010] The effective sub-image is moved back to its initial position in reverse, and the sub-corner points in the effective sub-image after the reverse movement are affinely transformed into the effective main image;

[0011] The depth of the target object in the image to be processed is determined based on the secondary camera angles in the effective secondary image derived from the affine transformation and the primary camera angles in the effective main image.

[0012] Optionally, determining the secondary corner point corresponding to each primary corner point in the secondary image to obtain a set of corner point pairs includes:

[0013] Based on the image quality of the main image and the sub-image, the number of optical flow pyramid layers is determined, and an optical flow pyramid is constructed based on the number of optical flow pyramid layers.

[0014] The optimal optical flow of the highest layer of the optical flow pyramid is calculated based on the light intensity of the main camera point in the main image and the light intensity of the secondary camera point in the secondary image.

[0015] Calculate the optical flow vector based on the optimal optical flow and the number of layers in the optical flow pyramid;

[0016] Based on the optical flow vector, the corresponding sub-corner point for each main camera corner point is determined in the sub-image, resulting in a set of corner point pairs.

[0017] Optionally, the step of affinely transforming the sub-corner points in the effective sub-image after reverse shifting to the effective main image includes:

[0018] The point matrix is ​​determined based on the correspondence between the corner points in the effective main image and the effective sub-image after reverse shifting;

[0019] Based on the point matrix, the sub-angle points in the effective sub-image after reverse shifting are affinely transformed to the effective main image.

[0020] Optionally, calculating the initial relative displacement between the main image and the sub-image in the image to be processed includes:

[0021] Remove image boundary effects from both the main image and the sub-image;

[0022] The main image after removing image boundary effects is transformed to obtain a frequency domain main image, and the sub-image after removing image boundary effects is transformed to obtain a frequency domain sub-image.

[0023] The cross-power spectrum is calculated based on the main frequency domain image, the conjugate of the main frequency domain image, and the sub-frequency domain image.

[0024] The initial relative displacement between the main image and the sub-image is calculated based on the cross-power spectrum.

[0025] Optionally, determining the mismatched corner pairs in the set of corner pairs includes:

[0026] According to the preset number of iterations, arbitrarily select a number of target corner point pairs corresponding to the fitting method from the set of corner point pairs;

[0027] The model is determined by constructing the number of interior points based on the selected target corner point pairs;

[0028] The unselected corner pairs in the corner pair set are input into the interior point number determination model to obtain the interior point number output by the interior point number determination model;

[0029] The model with the most interior points is selected as the target data model, and the exterior points determined by the target data model are used as the mismatched corner point pairs.

[0030] A second aspect of this disclosure provides a depth computing apparatus, comprising:

[0031] The calculation module is configured to calculate the initial relative displacement between the main image and the secondary image in the image to be processed, wherein the image to be processed is an image obtained by the stereo camera based on the target object, the main image is an image captured by the main camera of the stereo camera, and the secondary image is an image captured by the secondary camera of the stereo camera;

[0032] The moving module is configured to move the sub-image according to the initial relative displacement, such that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement.

[0033] The first determining module is configured to determine the main camera corner point in the main image and determine the sub-camera corner point corresponding to each main camera corner point in the sub-image, thereby obtaining a set of corner point pairs.

[0034] The deletion module is configured to identify mismatched corner pairs in the set of corner pairs and delete the mismatched corner pairs from the main image and the sub-image to obtain a valid main image and a valid sub-image.

[0035] The transformation module is configured to reverse-shift the effective sub-image back to its initial position and then affine-transform the sub-corner points in the reverse-shifted effective sub-image into the effective main image.

[0036] The second determining module is configured to determine the depth of the target object in the image to be processed based on the sub-angle points in the effective sub-image transformed into the effective main image and the main angle points in the effective main image.

[0037] Optionally, the first determining module is configured to:

[0038] Based on the image quality of the main image and the sub-image, the number of optical flow pyramid layers is determined, and an optical flow pyramid is constructed based on the number of optical flow pyramid layers.

[0039] The optimal optical flow of the highest layer of the optical flow pyramid is calculated based on the light intensity of the main camera point in the main image and the light intensity of the secondary camera point in the secondary image.

[0040] Calculate the optical flow vector based on the optimal optical flow and the number of layers in the optical flow pyramid;

[0041] Based on the optical flow vector, the corresponding sub-corner point for each main camera corner point is determined in the sub-image, resulting in a set of corner point pairs.

[0042] Optionally, the transformation module is configured as follows:

[0043] The point matrix is ​​determined based on the correspondence between the corner points in the effective main image and the effective sub-image after reverse shifting;

[0044] Based on the point matrix, the sub-angle points in the effective sub-image after reverse shifting are affinely transformed to the effective main image.

[0045] Optionally, the calculation module is configured to remove image boundary effects from the main image and the sub-image; and,

[0046] The main image, after removing image boundary effects, undergoes an image transformation to obtain a frequency domain main image; the secondary image, after removing image boundary effects, undergoes an image transformation to obtain a frequency domain secondary image; and...

[0047] Based on the frequency domain main image, the conjugate of the frequency domain main image, and the frequency domain sub-image, calculate the cross-power spectrum; and,

[0048] The initial relative displacement between the main image and the sub-image is calculated based on the cross-power spectrum.

[0049] Optionally, the deletion module is configured to select, according to a preset number of iterations, any number of target corner point pairs corresponding to the fitting method from the set of corner point pairs; and,

[0050] The model is determined by constructing the number of interior points based on the selected target corner point pairs; and,

[0051] The unselected corner pairs from the set of corner pairs are input into the interior point count determination model to obtain the interior point count output by the interior point count determination model; and,

[0052] The model with the most interior points is selected as the target data model, and the exterior points determined by the target data model are used as the mismatched corner point pairs.

[0053] According to a third aspect of the present disclosure, a computer apparatus is provided, comprising:

[0054] processor;

[0055] Memory used to store processor-executable instructions;

[0056] The processor is configured as follows:

[0057] Calculate the initial relative displacement between the main image and the secondary image in the image to be processed. The image to be processed is an image obtained by the stereo camera based on the target object. The main image is an image captured by the main camera of the stereo camera, and the secondary image is an image captured by the secondary camera of the stereo camera.

[0058] The sub-image is moved according to the initial relative displacement, such that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement.

[0059] Determine the main camera corner points in the main image, and determine the corresponding secondary camera corner points for each main camera corner point in the secondary image to obtain a set of corner point pairs;

[0060] Identify the mismatched corner pairs in the set of corner pairs, and delete the mismatched corner pairs from the main image and the sub-image to obtain a valid main image and a valid sub-image;

[0061] The effective sub-image is moved back to its initial position in reverse, and the sub-corner points in the effective sub-image after the reverse movement are affinely transformed into the effective main image;

[0062] The depth of the target object in the image to be processed is determined based on the secondary camera angles in the effective secondary image derived from the affine transformation and the primary camera angles in the effective main image.

[0063] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the method described in any one of the first aspects.

[0064] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0065] The process involves calculating the initial relative displacement between the main image and the sub-image in the image to be processed. The sub-image is then moved according to this initial relative displacement, ensuring that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement. The primary camera corners in the main image are determined, and the corresponding secondary camera corners in the sub-image are also determined, resulting in a set of corner point pairs. Mismatched corner point pairs are identified from this set and removed from both the main and sub-images, resulting in a valid main image and a valid sub-image. The valid sub-image is then moved back to its initial position, and the secondary camera corners in the reverse-moved valid sub-image are affinely transformed into the valid main image. Finally, the depth of the target object is determined based on the secondary camera corners in the valid sub-image transformed into the valid main image and the primary camera corners in the valid main image. In this way, corner matching is performed based on the moved sub-image, and mismatched corner pairs are identified and deleted. Then, the sub-corner points in the reverse-moved effective sub-image are affinely transformed to the effective main image. This can reduce the impact of image quality and repetitive texture areas, thereby improving the accuracy of determining the depth of the target object, and thus improving the accuracy of determining the spatial position of the target object.

[0066] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0067] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0068] Figure 1 This is a flowchart illustrating a depth calculation method according to an exemplary embodiment.

[0069] Figure 2 This is a schematic diagram illustrating a target object to be photographed according to an exemplary embodiment.

[0070] Figure 3 This is an implementation illustrated according to an exemplary embodiment. Figure 1 The flowchart for step S13.

[0071] Figure 4 This is an implementation illustrated according to an exemplary embodiment. Figure 1 The flowchart for step S15.

[0072] Figure 5 This is an implementation illustrated according to an exemplary embodiment. Figure 1 The flowchart for step S11.

[0073] Figure 6 This is an implementation illustrated according to an exemplary embodiment. Figure 1 The flowchart for step S14.

[0074] Figure 7 This is a block diagram illustrating a depth computing device according to an exemplary embodiment.

[0075] Figure 8 This is a block diagram illustrating an apparatus for a depth calculation method according to an exemplary embodiment. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0077] It should be noted that in this disclosure, the terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and should not be construed as describing a specific order or sequence.

[0078] In related technologies, the feature points of online calibration algorithms are easily affected by image quality, resulting in low accuracy. Furthermore, in areas with repetitive textures, they are prone to mismatches, leading to low accuracy in solving the homography matrix between the left and right images, and consequently, low accuracy in determining the depth of the target object.

[0079] In view of this, the present disclosure provides a depth calculation method that can effectively reduce the influence of image quality and repetitive texture regions, thereby improving the accuracy of determining the depth of a target object, and further improving the accuracy of determining the spatial location of the target object.

[0080] Figure 1 This is a flowchart illustrating a depth calculation method according to an exemplary embodiment. The method can be used in a terminal and includes the following steps.

[0081] In step S11, the initial relative displacement between the main image and the sub-image in the image to be processed is calculated.

[0082] Wherein, the image to be processed is an image obtained by the binocular camera based on the target object, the main image is an image captured by the main camera of the binocular camera, and the secondary image is an image captured by the secondary camera of the binocular camera.

[0083] In step S12, the sub-image is moved according to the initial relative displacement, so that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement.

[0084] In step S13, the main camera corner points in the main image are determined, and the corresponding secondary camera corner points in the secondary image are determined, thus obtaining a set of corner point pairs.

[0085] In step S14, mismatched corner pairs in the corner pair set are identified, and mismatched corner pairs are deleted from the main image and sub-image to obtain a valid main image and a valid sub-image.

[0086] In step S15, the effective sub-image is moved back to its initial position in reverse, and the sub-corner points in the effective sub-image after the reverse movement are affinely transformed into the effective main image.

[0087] In step S16, the depth of the target object in the image to be processed is determined based on the sub-angle points in the effective sub-image of the effective main image and the main angle points in the effective main image.

[0088] In practice, the main image is the image used for the final depth calculation, and the secondary image is the image to which the corner point simulation is transformed. For example... Figure 2 As shown, when the main camera is the left-eye camera and the right-eye camera is the secondary camera, the secondary camera's field of view (FOV) is wider than the main camera's. Therefore, the secondary image captured by the secondary camera has a larger field of view than the main image captured by the main camera. Thus, as... Figure 2 As shown in D, the sub-image and the main image will be misaligned due to the field of view. As a result, when the sub-image performs corner point matching with the main image, it is easy to match the wrong points.

[0089] See Figure 2 As shown, the width of the sub-image is wider than that of the main image. If the sub-image is adjusted by scaling, it may cause changes in the corner features of the sub-image, or even the loss of corner points, resulting in a higher error rate in corner point matching. Therefore, directly moving the sub-image to reduce the relative displacement between the main and sub-images can improve the accuracy of corner point matching.

[0090] For example, the initial relative displacement between the main image and the initial position of the sub-image in the image to be processed is calculated, and based on the initial relative displacement, the sub-image is moved from its initial position so that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement. Figure 2 Based on the initial relative displacement, the sub-image is moved to the right from its initial position.

[0091] Furthermore, based on the Shi-Tomasi algorithm, corner points are extracted from the main image according to the preset minimum feature value for corner detection and the preset maximum number of detected corner points. Specifically, based on the product between the minimum feature value and the maximum feature value in the image, and based on the maximum number of detected corner points, feature points with a minimum distance not less than a preset distance threshold are selected as the main camera corner points.

[0092] Furthermore, based on the initial relative displacement, the effective sub-image is moved back to the initial position in reverse. For example, if the sub-image is moved to the right by an initial relative displacement of (1,0) from the initial position, the effective sub-image is moved to the left by (-1,0) back to the initial position of the sub-image.

[0093] The above technical solution involves calculating the initial relative displacement between the main image and the sub-image in the image to be processed, moving the sub-image according to the initial relative displacement so that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement; determining the main camera corner points in the main image and the corresponding sub-camera corner points in the sub-image to obtain a set of corner point pairs; identifying mismatched corner point pairs in the set of corner point pairs and deleting them from the main image and sub-image to obtain a valid main image and a valid sub-image; moving the valid sub-image back to its initial position in reverse, and affine transforming the sub-camera corner points in the reverse-moved valid sub-image to the valid main image; and determining the depth of the target object based on the sub-camera corner points in the valid main image after the affine transformation and the main camera corner points in the valid main image. In this way, corner point matching is performed based on the moved sub-image, and mismatched corner point pairs are identified and deleted. Then, the sub-corner points in the reverse-moved effective sub-image are affinely transformed to the effective main image. This can improve the accuracy of feature point simulation transformation, reduce the influence of image quality and repetitive texture regions, thereby improving the accuracy of determining the depth of the target object, and thus improving the accuracy of determining the spatial position of the target object.

[0094] Based on the above embodiments, Figure 3 This is an implementation illustrated according to an exemplary embodiment. Figure 1 The flowchart of step S13 is as follows. In step S13, determining the sub-corner point corresponding to each main camera corner point in the sub-image to obtain a set of corner point pairs includes the following steps.

[0095] In step S131, the number of optical flow pyramid layers is determined based on the image quality of the main image and the sub-image, and an optical flow pyramid is constructed based on the number of optical flow pyramid layers.

[0096] In step S132, the optimal optical flow of the highest layer of the optical flow pyramid is calculated based on the light intensity of the main camera corner point in the main image and the light intensity of the secondary camera corner point in the secondary image.

[0097] In step S133, the optical flow vector is calculated based on the optimal optical flow and the number of optical flow pyramid layers.

[0098] In step S134, the sub-corner point corresponding to each main camera corner point is determined in the sub-image based on the optical flow vector, thus obtaining a set of corner point pairs.

[0099] Specifically, the main image is used as the current frame image, and the sub-image is used as the next frame image of the current frame. Assuming that the brightness of the pixels in the main image and the sub-image is the same and the motion is small, the sub-angle point corresponding to each main angle point can be determined in the sub-image based on the optical flow method.

[0100] Specifically, the number of optical flow pyramid layers is determined based on the pixels of the main image and the sub-image, and the number of optical flow pyramid layers is positively correlated with the image quality of the main image and the sub-image. For example, when the main image has 1000*750 pixels, the number of optical flow pyramid layers is determined to be 3. Usually, the number of optical flow pyramid layers does not exceed 5.

[0101] Furthermore, the image of the second-highest layer is pre-translated using the optimal optical flow of the highest layer, and the residual optical flow vector of the second-highest layer is calculated based on the optical flow of the second-highest layer. This process is repeated. In this way, the residual optical flow vector of each layer can be kept small.

[0102] For each layer of iterative optical flow calculation, the final optical flow vector is the superposition of the optical flows from all layers. Then, based on the optical flow vector, the corresponding sub-corner point is determined in the sub-image for each main camera corner point, resulting in corner point pairs, and a set of corner point pairs is obtained from these pairs.

[0103] The above technical solution, based on optical flow, uses the main image as the current frame image and the secondary image as the next frame image to determine corner pairs, which can improve the accuracy of corner pair matching and thus reduce the impact of image quality and repetitive texture areas.

[0104] Based on the above embodiments, Figure 4 This is an implementation illustrated according to an exemplary embodiment. Figure 1 The flowchart for step S15 is as follows. In step S15, the sub-angle points in the effective sub-image after reverse movement are affinely transformed into the effective main image, which includes the following steps.

[0105] In step S151, the point matrix is ​​determined based on the correspondence between the corner points in the effective main image and the effective sub-image after reverse movement.

[0106] In step S152, the sub-angle points in the effective sub-image after inverse movement are affinely transformed into the effective main image based on the point matrix.

[0107] In one implementation, the homography matrix between the effective main image and the effective sub-image after reverse shift can be determined based on the correspondence between the corner points in the effective main image and the effective sub-image after reverse shift. Then, based on the homography matrix, the sub-corner points in the effective sub-image after reverse shift are affinely transformed into the effective main image.

[0108] By adopting the above technical solution, the point matrix can be determined based on the correspondence between the corner points in the effective main image and the effective secondary image after reverse movement. This makes the homography matrix between the main and secondary images more accurate, optimizes the calibration and alignment results, and further improves the accuracy of determining the depth of the target object and the accuracy of determining the spatial position of the target object.

[0109] Based on the above embodiments, Figure 5 This is an implementation illustrated according to an exemplary embodiment. Figure 1 The flowchart of step S11 is as follows. In step S11, calculating the initial relative displacement between the main image and the sub-image in the image to be processed includes the following steps.

[0110] In step S111, image boundary effects of the main image and the sub-image are removed.

[0111] In step S112, the main image with the image boundary effect removed is transformed to obtain the main image in the frequency domain, and the sub-image with the image boundary effect removed is transformed to obtain the sub-image in the frequency domain.

[0112] In step S113, the cross power spectrum is calculated based on the main frequency domain image, the conjugate of the main frequency domain image, and the sub-frequency domain image.

[0113] In step S114, the initial relative displacement between the main image and the sub-image is calculated based on the cross power spectrum.

[0114] In practice, the initial relative displacement between the main image and the sub-image is calculated based on the phase correlation method. Specifically, the image boundary effects of the main image and the sub-image are removed using a Hanning window. Then, a Fourier transform is performed on the main image with the image boundary effects removed to obtain the frequency domain main image, and a Fourier transform is performed on the sub-image with the image boundary effects removed to obtain the frequency domain sub-image. The location of the maximum value of the cross power spectrum is determined, and a window of a preset size is established with this location as the center. The initial relative displacement between the main image and the sub-image is calculated within this window.

[0115] Based on the above embodiments, Figure 6 This is an implementation illustrated according to an exemplary embodiment. Figure 1 The flowchart for step S14 is as follows. In step S14, determining the mismatched corner pairs in the set of corner pairs includes the following steps.

[0116] In step S141, according to the preset number of iterations, the number of target corner point pairs corresponding to the fitting method is arbitrarily selected from the set of corner point pairs.

[0117] In step S142, the model is determined by constructing the number of interior points based on the selected target corner point pairs.

[0118] In step S143, the unselected corner pairs in the corner pair set are input into the interior point number determination model to obtain the interior point number output by the interior point number determination model.

[0119] In step S144, the model with the most interior points is selected as the target data model, and the exterior points determined by the target data model are taken as mismatched corner point pairs.

[0120] Optionally, when the fitting method is linear fitting, two target corner point pairs can be arbitrarily selected to construct an interior point number determination model; when the fitting method is homography matrix fitting, four target corner point pairs can be arbitrarily selected to construct an interior point number determination model.

[0121] After one iteration, the number of target corner point pairs corresponding to any fitting method is selected again to construct an interior point determination model until the number of fitting iterations reaches the preset number. In this way, based on the RANSAC algorithm, mismatched corner point pairs can be deleted, thereby improving the accuracy of matching the main camera corner point with the simulated transformation's secondary camera corner point.

[0122] Based on the same inventive concept, this disclosure also provides a depth computing device applied to a terminal for executing the steps of the depth computing method provided in the above embodiments. The device can implement the depth computing method in software, hardware, or a combination of both. Figure 7 This is a block diagram illustrating a depth computing device 100 according to an exemplary embodiment, such as... Figure 7 As shown, the device 100 includes: a calculation module 110, a movement module 120, a first determination module 130, a deletion module 140, a transformation module 150, and a second determination module 160.

[0123] The calculation module 110 is configured to calculate the initial relative displacement between the main image and the sub-image in the image to be processed. The image to be processed is an image obtained by the stereo camera based on the target object. The main image is an image captured by the main camera of the stereo camera, and the sub-image is an image captured by the sub-camera of the stereo camera.

[0124] The moving module 120 is configured to move the sub-image according to the initial relative displacement, such that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement.

[0125] The first determining module 130 is configured to determine the main camera corner point in the main image and determine the sub-camera corner point corresponding to each main camera corner point in the sub-image, thereby obtaining a set of corner point pairs;

[0126] The deletion module 140 is configured to determine mismatched corner pairs in the corner pair set and delete the mismatched corner pairs from the main image and the sub-image to obtain a valid main image and a valid sub-image.

[0127] The transformation module 150 is configured to reverse-move the effective sub-image back to its initial position, and to affine transform the sub-corner points in the reverse-moved effective sub-image into the effective main image;

[0128] The second determining module 160 is configured to determine the depth of the target object in the image to be processed based on the sub-angle points in the effective sub-image transformed into the effective main image and the main angle points in the effective main image.

[0129] The aforementioned device performs corner matching based on the moved sub-image, identifies and deletes mismatched corner pairs, and then performs an affine transformation on the sub-corner points in the reverse-moved effective sub-image to the effective main image. This can reduce the impact of image quality and repetitive texture areas, thereby improving the accuracy of determining the depth of the target object, and further improving the accuracy of determining the spatial position of the target object.

[0130] Optionally, the first determining module 130 is configured to determine the number of optical flow pyramid layers based on the image quality of the main image and the sub-image, and to construct an optical flow pyramid based on the number of optical flow pyramid layers;

[0131] The optimal optical flow of the highest layer of the optical flow pyramid is calculated based on the light intensity of the main camera point in the main image and the light intensity of the secondary camera point in the secondary image.

[0132] Calculate the optical flow vector based on the optimal optical flow and the number of layers in the optical flow pyramid;

[0133] Based on the optical flow vector, the corresponding sub-corner point for each main camera corner point is determined in the sub-image, resulting in a set of corner point pairs.

[0134] Optionally, the transformation module 150 is configured to determine a point matrix based on the correspondence between corner points in the effective main image and the reverse-shifted effective sub-image;

[0135] Based on the point matrix, the sub-angle points in the effective sub-image after reverse shifting are affinely transformed to the effective main image.

[0136] Optionally, the calculation module 110 is configured to remove image boundary effects from the main image and the sub-image; and,

[0137] The main image, after removing image boundary effects, undergoes an image transformation to obtain a frequency domain main image; the secondary image, after removing image boundary effects, undergoes an image transformation to obtain a frequency domain secondary image; and...

[0138] Based on the frequency domain main image, the conjugate of the frequency domain main image, and the frequency domain sub-image, calculate the cross-power spectrum; and,

[0139] The initial relative displacement between the main image and the sub-image is calculated based on the cross-power spectrum.

[0140] Optionally, the deletion module 140 is configured to arbitrarily select a number of target corner point pairs corresponding to the fitting method from the corner point pair set according to a preset number of iterations; and,

[0141] The model is determined by constructing the number of interior points based on the selected target corner point pairs; and,

[0142] The unselected corner pairs from the set of corner pairs are input into the interior point count determination model to obtain the interior point count output by the interior point count determination model; and,

[0143] The model with the most interior points is selected as the target data model, and the exterior points determined by the target data model are used as the mismatched corner point pairs.

[0144] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0145] Furthermore, it is worth noting that, for the sake of convenience and brevity, the embodiments described in the specification are all preferred embodiments, and the parts involved are not necessarily essential to this disclosure. For example, the transformation module 150 and the second determining module 160 can be independent devices or the same device in specific implementations, and this disclosure does not limit them.

[0146] According to embodiments of this disclosure, a computer device is also provided, comprising:

[0147] processor;

[0148] Memory used to store processor-executable instructions;

[0149] The processor is configured as follows:

[0150] Calculate the initial relative displacement between the main image and the secondary image in the image to be processed. The image to be processed is an image obtained by the stereo camera based on the target object. The main image is an image captured by the main camera of the stereo camera, and the secondary image is an image captured by the secondary camera of the stereo camera.

[0151] The sub-image is moved according to the initial relative displacement, such that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement.

[0152] Determine the main camera corner points in the main image, and determine the corresponding secondary camera corner points for each main camera corner point in the secondary image to obtain a set of corner point pairs;

[0153] Identify the mismatched corner pairs in the set of corner pairs, and delete the mismatched corner pairs from the main image and the sub-image to obtain a valid main image and a valid sub-image;

[0154] The effective sub-image is moved back to its initial position in reverse, and the sub-corner points in the effective sub-image after the reverse movement are affinely transformed into the effective main image;

[0155] The depth of the target object in the image to be processed is determined based on the secondary camera angles in the effective secondary image derived from the affine transformation and the primary camera angles in the effective main image.

[0156] According to embodiments of this disclosure, a computer-readable storage medium is also provided, on which computer program instructions are stored, which, when executed by a processor, implement the steps in any of the aforementioned depth computing methods.

[0157] Figure 8 This is a block diagram illustrating an apparatus 800 for a depth computing method according to an exemplary embodiment. Optionally, the apparatus 800 can be configured as a computer device for performing the steps of the aforementioned depth computing method. For example, the apparatus 800 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0158] Reference Figure 8 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0159] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the aforementioned depth calculation method. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0160] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0161] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 800.

[0162] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0163] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as a shooting mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0164] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0165] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0166] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0167] In an exemplary embodiment, the device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described deep computing method.

[0168] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to complete the aforementioned depth calculation method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0169] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described depth calculation method when executed by the programmable device.

[0170] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0171] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A depth calculation method, characterized in that, include: Calculate the initial relative displacement between the main image and the secondary image in the image to be processed. The image to be processed is an image obtained by the stereo camera based on the target object. The main image is an image captured by the main camera of the stereo camera, and the secondary image is an image captured by the secondary camera of the stereo camera. The sub-image is moved according to the initial relative displacement, such that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement. Determine the main camera corner points in the main image, and determine the corresponding secondary camera corner points for each main camera corner point in the secondary image to obtain a set of corner point pairs; Identify the mismatched corner pairs in the set of corner pairs, and delete the mismatched corner pairs from the main image and the sub-image to obtain a valid main image and a valid sub-image; The effective sub-image is moved back to its initial position in reverse, and the sub-corner points in the effective sub-image after the reverse movement are affinely transformed into the effective main image; The depth of the target object in the image to be processed is determined based on the secondary camera angles in the effective secondary image derived from the affine transformation and the primary camera angles in the effective main image.

2. The method according to claim 1, characterized in that, The step of determining the secondary camera corner point corresponding to each primary camera corner point in the secondary image to obtain a set of corner point pairs includes: Based on the image quality of the main image and the sub-image, the number of optical flow pyramid layers is determined, and an optical flow pyramid is constructed based on the number of optical flow pyramid layers. The optimal optical flow of the highest layer of the optical flow pyramid is calculated based on the light intensity of the main camera point in the main image and the light intensity of the secondary camera point in the secondary image. Calculate the optical flow vector based on the optimal optical flow and the number of layers in the optical flow pyramid; Based on the optical flow vector, the corresponding sub-corner point for each main camera corner point is determined in the sub-image, resulting in a set of corner point pairs.

3. The method according to claim 1, characterized in that, The step of affinely transforming the sub-corner points in the effective sub-image after reverse shifting to the effective main image includes: The point matrix is ​​determined based on the correspondence between the corner points in the effective main image and the effective sub-image after reverse shifting; Based on the point matrix, the sub-angle points in the effective sub-image after reverse shifting are affinely transformed to the effective main image.

4. The method according to claim 1, characterized in that, The calculation of the initial relative displacement between the main image and the sub-image in the image to be processed includes: Remove image boundary effects from both the main image and the sub-image; The main image after removing image boundary effects is transformed to obtain a frequency domain main image, and the sub-image after removing image boundary effects is transformed to obtain a frequency domain sub-image. The cross-power spectrum is calculated based on the main frequency domain image, the conjugate of the main frequency domain image, and the sub-frequency domain image. The initial relative displacement between the main image and the sub-image is calculated based on the cross-power spectrum.

5. The method according to claim 1, characterized in that, Determining the mismatched corner pairs in the set of corner pairs includes: According to the preset number of iterations, arbitrarily select a number of target corner point pairs corresponding to the fitting method from the set of corner point pairs; The model is determined by constructing the number of interior points based on the selected target corner point pairs; The unselected corner pairs in the corner pair set are input into the interior point number determination model to obtain the interior point number output by the interior point number determination model; The model with the most interior points is selected as the target data model, and the exterior points determined by the target data model are used as the mismatched corner point pairs.

6. A depth computing device, characterized in that, include: The calculation module is configured to calculate the initial relative displacement between the main image and the secondary image in the image to be processed, wherein the image to be processed is an image obtained by the stereo camera based on the target object, the main image is an image captured by the main camera of the stereo camera, and the secondary image is an image captured by the secondary camera of the stereo camera; The moving module is configured to move the sub-image according to the initial relative displacement, such that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement. The first determining module is configured to determine the main camera corner point in the main image and determine the sub-camera corner point corresponding to each main camera corner point in the sub-image, thereby obtaining a set of corner point pairs. The deletion module is configured to identify mismatched corner pairs in the set of corner pairs and delete the mismatched corner pairs from the main image and the sub-image to obtain a valid main image and a valid sub-image. The transformation module is configured to reverse-shift the effective sub-image back to its initial position and then affine-transform the sub-corner points in the reverse-shifted effective sub-image into the effective main image. The second determining module is configured to determine the depth of the target object in the image to be processed based on the sub-angle points in the effective sub-image transformed into the effective main image and the main angle points in the effective main image.

7. The apparatus according to claim 6, characterized in that, The first determining module is configured as follows: Based on the image quality of the main image and the sub-image, the number of optical flow pyramid layers is determined, and an optical flow pyramid is constructed based on the number of optical flow pyramid layers. The optimal optical flow of the highest layer of the optical flow pyramid is calculated based on the light intensity of the main camera point in the main image and the light intensity of the secondary camera point in the secondary image. Calculate the optical flow vector based on the optimal optical flow and the number of layers in the optical flow pyramid; Based on the optical flow vector, the corresponding sub-corner point for each main camera corner point is determined in the sub-image, resulting in a set of corner point pairs.

8. The apparatus according to claim 6, characterized in that, The transformation module is configured as follows: The point matrix is ​​determined based on the correspondence between the corner points in the effective main image and the effective sub-image after reverse shifting; Based on the point matrix, the sub-angle points in the effective sub-image after reverse shifting are affinely transformed to the effective main image.

9. A computer device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured as follows: Calculate the initial relative displacement between the main image and the secondary image in the image to be processed. The image to be processed is an image obtained by the stereo camera based on the target object. The main image is an image captured by the main camera of the stereo camera, and the secondary image is an image captured by the secondary camera of the stereo camera. The sub-image is moved according to the initial relative displacement, such that the relative displacement between the main image and the moved sub-image is less than the initial relative displacement. Determine the main camera corner points in the main image, and determine the corresponding secondary camera corner points for each main camera corner point in the secondary image to obtain a set of corner point pairs; Identify the mismatched corner pairs in the set of corner pairs, and delete the mismatched corner pairs from the main image and the sub-image to obtain a valid main image and a valid sub-image; The effective sub-image is moved back to its initial position in reverse, and the sub-corner points in the effective sub-image after the reverse movement are affinely transformed into the effective main image; The depth of the target object in the image to be processed is determined based on the secondary camera angles in the effective secondary image derived from the affine transformation and the primary camera angles in the effective main image.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 1 to 5.

Images