Stereoscopic imaging method, device, equipment and storage medium

Abstract

The invention provides a stereo imaging method, device and equipment and a storage medium, the method is applied to stereo imaging equipment comprising cameras with different focal lengths, and the method comprises the following steps: obtaining a first short-focus image and a first long-focus image at a first moment, and a second short-focus image and a second long-focus image at a second moment; obtaining an area which is not overlapped with the first short-focus image and the second long-focus image and is overlapped with the first long-focus image in the second short-focus image; for each first pixel point in the area, determining the depth corresponding to the first pixel point by utilizing a depth relational expression based on the moving distance of the stereoscopic imaging equipment from the first moment to the second moment, a first imaging position corresponding to the first pixel point in the short-focus camera and a second imaging position corresponding to the second pixel point in the long-focus camera; and outputting a target stereo image corresponding to the second moment according to the depth corresponding to each first pixel point. The three-dimensional imaging range of cameras with different focal lengths can be expanded.

Description

Technical Field

[0001] This application relates to the field of visual perception technology, and particularly relates to a stereoscopic imaging method, device, equipment and storage medium. Background Art

[0002] Visual perception is a key technology in the intelligent driving industry. To enable a vehicle to perceive the driving environment, it is mostly necessary to perform target detection and stereoscopic imaging on the driving environment based on images collected by a camera. Currently, cameras with the same focal length are mostly used, such as a binocular camera with two identical focal lengths. Based on the images collected by the cameras with the same focal length, close-range targets can be detected, but the detection distance of the cameras with the same focal length is limited, resulting in the inability to detect distant targets.

[0003] Compared with cameras with the same focal length, the telephoto camera in cameras with different focal lengths (such as a binocular camera with different focal lengths) has a longer detection distance. Based on the images collected by the cameras with different focal lengths, not only can close-range targets be detected, but also distant targets can be detected. However, currently, the stereoscopic imaging range of cameras with different focal lengths is limited, mostly confined to the overlapping area of the short-focus image and the long-focus image, which affects its use effect. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a stereoscopic imaging method, device, equipment and storage medium, aiming to expand the stereoscopic imaging range of cameras with different focal lengths.

[0005] The present application provides a three-dimensional imaging method, which is applied to a three-dimensional imaging device. The three-dimensional imaging device includes a short-focus camera and a long-focus camera. The three-dimensional imaging method includes: acquiring a first short-focus image and a second short-focus image collected by the short-focus camera at a first moment and a second moment respectively, and a first long-focus image and a second long-focus image collected by the long-focus camera at the first moment and the second moment respectively; wherein, the first moment is the previous image acquisition moment of the second moment; acquiring a first target area; wherein, the first target area is an area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image, and overlaps with the first long-focus image; for each first pixel point in the first target area, based on the first moving distance of the three-dimensional imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera, using a first depth relationship formula, determining the depth corresponding to the first pixel point; the second pixel point is the pixel point in the first long-focus image that matches the first pixel point; wherein, the first depth relationship formula is determined based on the third imaging position of a first spatial point in the long-focus camera at a third moment, and the fourth imaging position of the first spatial point in the short-focus camera at a fourth moment, using the principle of similar triangles and the principle of parallax; the third moment is the previous image acquisition moment of the fourth moment; according to the depth corresponding to each first pixel point, outputting a target three-dimensional image corresponding to the second moment.

[0006] In an embodiment, before determining the depth corresponding to the first pixel point by using a first depth relationship formula based on the first moving distance of the stereoscopic imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera, the stereoscopic imaging method further includes: acquiring the first spatial point, the third pixel point corresponding to the third imaging position, the fourth pixel point corresponding to the fourth imaging position, the first optical center point of the short-focus camera at the third moment, the second optical center point of the short-focus camera at the fourth moment, and the third optical center point of the long-focus camera at the third moment; determining the intersection point of the imaging plane of the short-focus camera at the fourth moment and a first straight line as a first target point; the first straight line is a straight line passing through the second optical center point and parallel to a second straight line; the second straight line is a straight line passing through the first spatial point and the third optical center point; determining the intersection point of a third straight line and the second straight line as a second target point; the third straight line is a straight line passing through the fourth pixel point and parallel to the imaging plane of the short-focus camera at the fourth moment; determining the intersection point of a fourth straight line and a fifth straight line as a third target point; the fourth straight line is a straight line passing through the first optical center point and parallel to the imaging plane of the short-focus camera at the third moment; the fifth straight line is a straight line passing through the first spatial point and the second optical center point; determining the intersection point of the first straight line and the fourth straight line as a fourth target point; determining the intersection point of the fourth straight line and the second straight line as a fifth target point; using the principle of similar triangles, based on the third pixel point, the fourth pixel point, the third target point, the fifth target point, the first optical center point, the second optical center point, and the third optical center point, determining a first distance relationship formula; the first distance relationship formula is used to solve the distance between the third target point and the fifth target point; using the principle of similar triangles, based on the third pixel point, the fourth pixel point, the first optical center point, the second optical center point, the third optical center point, the first target point, the fourth target point, and the fifth target point, determining a second distance relationship formula; the second distance relationship formula is used to solve the distance between the fourth pixel point and the second target point; using the parallax principle, based on the first distance relationship formula and the second distance relationship formula, determining the first depth relationship formula.

[0007] In an embodiment, the first depth relationship formula is:

[0008]

[0009] where t1 is the fourth moment, Z t1is the depth of the first spatial point relative to the second optical center point of the short-focus camera at the fourth moment; B is the baseline length between the short-focus camera and the long-focus camera; f l is the focal length of the long-focus camera; f s is the focal length of the short-focus camera; S is the second moving distance of the stereoscopic imaging device from the third moment to the fourth moment; x1 is the horizontal distance between the third pixel point corresponding to the third imaging position and the second center point of the imaging plane of the long-focus camera at the third moment, and x3 is the horizontal distance between the fourth pixel point corresponding to the fourth imaging position and the first center point of the imaging plane of the short-focus camera at the fourth moment.

[0010] In an embodiment, the outputting the target stereoscopic image corresponding to the second moment according to the depth corresponding to each of the first pixel points includes: determining the first stereoscopic image corresponding to the first target region at the second moment according to the depth corresponding to each of the first pixel points; obtaining a second target region; the second target region is a region in the second short-focus image that does not overlap with the first short-focus image and overlaps with the second long-focus image; for each fifth pixel point in the second target region, determining the depth corresponding to the fifth pixel point according to the fifth imaging position of the fifth pixel point corresponding to the short-focus camera and the sixth imaging position of the sixth pixel point corresponding to the long-focus camera, using a second depth relationship; the sixth pixel point is the pixel point in the second long-focus image that matches the fifth pixel point; the second depth relationship is determined based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth moment and the eighth imaging position of the second spatial point in the long-focus camera at the fifth moment, using the principle of similar triangles and the principle of parallax; determining the second stereoscopic image corresponding to the second target region at the second moment according to the depth corresponding to each of the fifth pixel points; and outputting the target stereoscopic image according to the first stereoscopic image and the second stereoscopic image.

[0011] In one embodiment, outputting the target stereoscopic image corresponding to the second moment according to the depth corresponding to each of the first pixel points includes: determining a first stereoscopic image corresponding to the first target region at the second moment according to the depth corresponding to each of the first pixel points; obtaining a third target region; the third target region is the region in the first short-focus image that overlaps with the first long-focus image; for each seventh pixel point in the third target region, determining the depth corresponding to the seventh pixel point according to the ninth imaging position corresponding to the seventh pixel point and the short-focus camera and the tenth imaging position corresponding to the eighth pixel point and the long-focus camera by using a second depth relationship; the eighth pixel point is the pixel point in the first long-focus image that matches the seventh pixel point; the second depth relationship is determined by using the principle of similar triangles and the principle of parallax based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth moment and the eighth imaging position of the second spatial point in the long-focus camera at the fifth moment; determining a third stereoscopic image corresponding to the third target region at the first moment according to the depth corresponding to each of the seventh pixel points; and outputting the target stereoscopic image according to the first stereoscopic image and the third stereoscopic image.

[0012] In one embodiment, outputting the stereoscopic image corresponding to the second moment according to the depth corresponding to each of the first pixel points includes: for each of the first pixel points, determining the spatial coordinates corresponding to the first pixel point by using the principle of similar triangles based on the depth corresponding to the first pixel point, the first imaging position corresponding to the first pixel point and the short-focus camera, and the focal length of the short-focus camera; and outputting the target stereoscopic image according to the spatial coordinates corresponding to each of the first pixel points.

[0013] In one embodiment, obtaining the first target region includes: respectively obtaining the spatial target categories corresponding to the pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image; using a pixel matching algorithm to match the pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image according to the spatial target categories corresponding to the pixel points to obtain a pixel matching result; and determining the first target region according to the pixel matching result.

[0014] An embodiment of the present application further provides a stereoscopic imaging device. The stereoscopic imaging device includes a first acquisition module, a second acquisition module, a depth determination module, and a stereoscopic imaging module. The first acquisition module is configured to acquire a first short-focus image and a second short-focus image collected by a short-focus camera at a first moment and a second moment respectively, and a first long-focus image and a second long-focus image collected by a long-focus camera at the first moment and the second moment respectively. Wherein, the first moment is the previous image acquisition moment of the second moment. The second acquisition module is configured to acquire a first target area. Wherein, the first target area is an area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image, and overlaps with the first long-focus image. The depth determination module is configured to, for each first pixel point in the first target area, based on a first moving distance of the stereoscopic imaging device from the first moment to the second moment, a first imaging position corresponding to the first pixel point in the short-focus camera, and a second imaging position corresponding to a second pixel point in the long-focus camera, use a first depth relationship formula to determine the depth corresponding to the first pixel point. The second pixel point is a pixel point in the first long-focus image that matches the first pixel point. Wherein, the first depth relationship formula is determined based on a third imaging position of a first spatial point in the long-focus camera at a third moment, a fourth imaging position of the first spatial point in the short-focus camera at a fourth moment, using the principle of similar triangles and the principle of parallax. The third moment is the previous image acquisition moment of the fourth moment. The stereoscopic imaging module is configured to output a target stereoscopic image corresponding to the second moment according to the depth corresponding to each first pixel point.

[0015] An embodiment of the present application further provides a stereoscopic imaging device. The stereoscopic imaging device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the above stereoscopic imaging method is implemented.

[0016] An embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above stereoscopic imaging method is implemented.

[0017] A stereoscopic imaging method, apparatus, device, and storage medium provided by the present application determine an area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image but overlaps with the first long-focus image. For each first pixel point in this area, based on the first moving distance of the stereoscopic imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera, using the first depth relationship formula, the depth corresponding to the first pixel point is determined, and based on the depth corresponding to the first pixel point, the target stereoscopic image corresponding to the second moment is output, enabling stereoscopic imaging of the non-overlapping area between the short-focus image and the long-focus image collected at the same moment, thereby expanding the stereoscopic imaging range of cameras with different focal lengths. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of the stereoscopic imaging method provided by an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the determination process of the first depth relationship formula provided by an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the determination process of the second depth relationship formula provided by an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of the specific process of the stereoscopic imaging method provided by an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the structure of the stereoscopic imaging apparatus provided by an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of the structure of the stereoscopic imaging device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Next, the technical solutions in the embodiments of the present application will be clearly described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application belong to the scope of protection of the present application.

[0025] The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / ", generally represents an "or" relationship between the associated objects before and after.

[0026] The three-dimensional imaging method provided by the embodiments of this application can be applied to a three-dimensional imaging device, which can be a vehicle or an electronic device. In some embodiments, the electronic device can be a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Further, the three-dimensional imaging method provided by the embodiments of this application can also be applied to the software of the three-dimensional imaging device, and the software can be an application that implements the three-dimensional imaging method, etc., but is not limited to the above forms.

[0027] The following will combine the accompanying drawings and specifically illustrate the three-dimensional imaging method provided by the embodiments of this application through specific embodiments.

[0028] A three-dimensional imaging method provided by the embodiments of this application is applied to a three-dimensional imaging device, which includes a short-focus camera and a long-focus camera. Please refer to Figure 1 , a three-dimensional imaging method provided by the embodiments of this application may include:

[0029] Step S101: Obtain a first short-focus image and a second short-focus image collected by the short-focus camera at a first moment and a second moment respectively, and a first long-focus image and a second long-focus image collected by the long-focus camera at the first moment and the second moment respectively; wherein, the first moment is the previous image acquisition moment of the second moment;

[0030] Step S102: Obtain a first target area; wherein, the first target area is an area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image, and overlaps with the first long-focus image;

[0031] Step S103: For each first pixel point in the first target area, based on the first moving distance of the stereoscopic imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera, use the first depth relationship formula to determine the depth corresponding to the first pixel point; the second pixel point is the pixel point in the first long-focus image that matches the first pixel point; wherein, the first depth relationship formula is determined based on the third imaging position of the first spatial point in the long-focus camera at the third moment and the fourth imaging position of the first spatial point in the short-focus camera at the fourth moment, using the principle of similar triangles and the principle of parallax; the third moment is the previous image acquisition moment of the fourth moment.

[0032] Step S104: Output the target stereoscopic image corresponding to the second moment according to the depth corresponding to each first pixel point.

[0033] In the embodiment of the present application, by determining the area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image and overlaps with the first long-focus image, and for each first pixel point in this area, based on the first moving distance of the stereoscopic imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera, use the first depth relationship formula to determine the depth corresponding to the first pixel point, and based on the depth corresponding to the first pixel point, output the target stereoscopic image corresponding to the second moment, it is possible to perform stereoscopic imaging on the non-overlapping area in the short-focus image and the long-focus image collected at the same moment, thereby expanding the stereoscopic imaging range of cameras with different focal lengths.

[0034] Optionally, the above first short-focus image is the image collected by the short-focus camera at the first moment, the above second short-focus image is the image collected by the short-focus camera at the second moment, the above first long-focus image is the image collected by the long-focus camera at the first moment, and the above second long-focus image is the image collected by the long-focus camera at the second moment.

[0035] In an implementation manner, the obtaining the first target area in the above step S102 includes:

[0036] Obtain the spatial target categories corresponding to the pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image respectively;

[0037] Use the pixel matching algorithm to match the pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image according to the spatial target categories corresponding to the pixel points to obtain a pixel matching result;

[0038] Determine the first target area according to the pixel matching result.

[0039] Optionally, the spatial target category is the classification of the target object in the three-dimensional space, such as pedestrians, vehicles, manhole covers, speed bumps, curbs, and potholes. The spatial target category corresponding to the pixel point can be determined based on the features of the pixel point, and the features of the pixel point can include the color, texture, and shape of the pixel point. The pixel matching algorithm can include at least one of a convolutional neural network, a multi-scale feature extraction network, a disparity-guided network, and a generative adversarial network.

[0040] Further, in the case of obtaining the pixel matching result, the area where the pixel points in the second short-focus image do not match the pixel points in the first short-focus image and the pixel points in the second long-focus image, but match the pixel points in the first long-focus image can be determined as the first target area.

[0041] The embodiment of the present application uses a pixel matching algorithm to match the pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image based on the spatial target category corresponding to the pixel points, and can obtain an accurate pixel matching result, thereby improving the accuracy of determining the first target area.

[0042] In another implementation manner, obtaining the first target area in the above step S102 includes:

[0043] Obtain the first field of view angle area of the short-focus camera and the second field of view angle area of the long-focus camera at the first moment, and the third field of view angle area of the short-focus camera and the fourth field of view angle area of the long-focus camera at the second moment;

[0044] The area in the third field of view angle area that does not overlap with the first field of view angle area and the fourth field of view angle area, but overlaps with the second field of view angle area is determined as the fifth field of view angle area;

[0045] The area in the second short-focus image corresponding to the fifth field of view angle area is determined as the first target area.

[0046] Based on the overlapping situation of the field of view angles of the long-focus camera and the short-focus camera at different moments, the embodiment of the present application can quickly and accurately determine the fifth field of view angle area in the third field of view angle area that does not overlap with the first field of view angle area and the fourth field of view angle area, but overlaps with the second field of view angle area. Further, by matching the fifth field of view angle area with the area in the second short-focus image, the first target area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image, but overlaps with the first long-focus image can be quickly and accurately determined.

[0047] Optionally, the above first depth relationship can be a relationship pre-stored in the stereoscopic imaging device or a relationship derived instantaneously. The above third moment and the above first moment can be the same moment or different moments, and the above fourth moment and the above second moment can be the same moment or different moments; among them, when the third moment and the first moment are different moments, and the fourth moment and the second moment are different moments, the above first depth relationship is a relationship pre-stored in the stereoscopic imaging device, the third moment is a historical moment earlier than the first moment, and the fourth moment is a historical moment earlier than the second moment. It can be understood that in this case, the first depth relationship is a fixed formula and can be directly called when actually needed; when the third moment and the first moment are the same moment, and the fourth moment and the second moment are the same moment, the above first depth relationship is a relationship instantaneously derived by the stereoscopic imaging device. It can be understood that in this case, the first depth relationship is a non-fixed formula and can be instantaneously derived when actually needed.

[0048] Optionally, the first depth relationship is a relationship among the depth of the first spatial point relative to the optical center of the short-focus camera at the fourth moment, the baseline length between the short-focus camera and the long-focus camera, the focal length of the long-focus camera, the focal length of the short-focus camera, the second moving distance of the stereoscopic imaging device from the third moment to the fourth moment, the lateral distance between the third pixel point corresponding to the third imaging position and the second center point of the imaging plane of the long-focus camera at the third moment, and the lateral distance between the fourth pixel point corresponding to the fourth imaging position and the first center point of the imaging plane of the short-focus camera at the fourth moment.

[0049] Furthermore, the first moving distance of the stereoscopic imaging device from the first moment to the second moment, the lateral distance corresponding to the first imaging position, the lateral distance corresponding to the second imaging position, the baseline length between the short-focus camera and the long-focus camera, the focal length of the long-focus camera, and the focal length of the short-focus camera can be substituted into the first depth relationship to obtain the depth corresponding to the first pixel point. Optionally, the depth corresponding to the first pixel point represents the depth of the spatial point corresponding to the first pixel point relative to the optical center point of the short-focus camera at the second moment.

[0050] In an embodiment, before determining the depth corresponding to the first pixel point by using the first depth relationship based on the first moving distance of the stereoscopic imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera in step S103, the stereoscopic imaging method provided by the embodiments of the present application further includes:

[0051] Obtaining a first spatial point, a third pixel point corresponding to a third imaging position, a fourth pixel point corresponding to a fourth imaging position, a first optical center point of the short-focus camera at the third moment, a second optical center point of the short-focus camera at the fourth moment, and a third optical center point of the long-focus camera at the third moment;

[0052] Determine the intersection point of the imaging plane of the short - focus camera at the fourth moment and the first straight line as the first target point; the first straight line is a straight line passing through the second optical center point and parallel to the second straight line; the second straight line is a straight line passing through the first spatial point and the third optical center point.

[0053] Determine the intersection point of the third straight line and the second straight line as the second target point; the third straight line is a straight line passing through the fourth pixel point and parallel to the imaging plane of the short - focus camera at the fourth moment.

[0054] Determine the intersection point of the fourth straight line and the fifth straight line as the third target point; the fourth straight line is a straight line passing through the first optical center point and parallel to the imaging plane of the short - focus camera at the third moment; the fifth straight line is a straight line passing through the first spatial point and the second optical center point.

[0055] Determine the intersection point of the first straight line and the fourth straight line as the fourth target point;

[0056] Determine the intersection point of the fourth straight line and the second straight line as the fifth target point;

[0057] Using the principle of similar triangles, based on the third pixel point, the fourth pixel point, the third target point, the fifth target point, the first optical center point, the second optical center point, and the third optical center point, determine the first distance relationship; the first distance relationship is used to solve for the distance between the third target point and the fifth target point.

[0058] Using the principle of similar triangles, based on the third pixel point, the fourth pixel point, the first optical center point, the second optical center point, the third optical center point, the first target point, the fourth target point, and the fifth target point, determine the second distance relationship; the second distance relationship is used to solve for the distance between the fourth pixel point and the second target point.

[0059] Using the parallax principle, based on the first distance relationship and the second distance relationship, determine the first depth relationship.

[0060] Please refer to Figure 2 , the Field of View (FOV) region of the short - focus camera at the third moment is the 1# sector area, the FOV region of the long - focus camera at the third moment is the 2# sector area, the FOV region of the short - focus camera at the fourth moment is the 3# sector area, and the FOV region of the long - focus camera at the fourth moment is the 4# sector area. From Figure 2It can be known that when the three-dimensional imaging device moves forward by a second moving distance S from the third moment to the fourth moment, the telephoto camera and the short-focus camera also move forward synchronously by the second moving distance S. In the FOV area of the short-focus camera at the fourth moment, there are areas D1 and D3 that do not overlap with the FOV area of the short-focus camera at the third moment and the FOV area of the telephoto camera at the fourth moment, but overlap with the FOV area of the telephoto camera at the third moment. Correspondingly, the image areas corresponding to areas D1 and D3 in the image collected by the short-focus camera at the fourth moment do not overlap with the image collected by the short-focus camera at the third moment and the image collected by the telephoto camera at the fourth moment, but overlap with the image collected by the telephoto camera at the third moment.

[0061] Please continue to refer to Figure 2 , the first target space point a can be any space point in areas D1 and D3. The fourth pixel point c, the second optical center point e, and the first center point g of the imaging plane of the short-focus camera at the fourth moment form a triangle △ecg. The second optical center point e, the first optical center point d, and the third target point 3 form a triangle △ed3. The triangle △ecg and the triangle △ed3 are similar triangles.

[0062] The intersection point n of the fourth straight line and the sixth straight line, the third optical center point f, and the fifth target point 5 form a triangle △nf5. The third pixel point b, the second center point m of the imaging plane of the telephoto camera at the third moment, and the third optical center point f form a triangle △bmf. The triangle △nf5 and the triangle △bmf are similar triangles. Among them, the sixth straight line is the straight line passing through the second center point m and the third optical center point f.

[0063] The principle of similar triangles can be used, combined with formulas (1) to (3), to determine the first distance relationship:

[0064]

[0065] A = B - x6 + x5 (3)

[0066] Among them, x1 is the horizontal distance between the third pixel point b and the second center point m, x3 is the horizontal distance between the fourth pixel point c and the first center point g, x5 is the horizontal distance between the third target point 3 and the first optical center point d, x6 is the horizontal distance between the fifth target point 5 and the intersection point n, f l is the focal length of the telephoto camera, f s is the focal length of the short-focus camera, S is the second moving distance of the three-dimensional imaging device from the third moment to the fourth moment, B is the baseline distance between the short-focus camera and the telephoto camera, and A is the horizontal distance between the third target point 3 and the fifth target point 5.

[0067] Please continue to refer to Figure 2, the first target point 1, the first center point g, and the second optical center point e form a triangle △ge1. △ge1 and △bmf are similar triangles. The second optical center point e, the first optical center point d, and the fourth target point 4 form △ed4. △ed4 and △ge1 are similar triangles. The first target point 1 and the second target point 2 form a line segment L12. The fourth target point 4 and the fifth target point 5 form a line segment L45. The length of the line segment L12 is equal to the length of the line segment L45.

[0068] Using the principle of similar triangles and combining formulas (1), (4) - (8), the second distance relationship can be determined:

[0069]

[0070] x7 = B - x2 - x6 (6)

[0071] D = x4 + x3 (7)

[0072] x8 = x7 - D (8)

[0073] Among them, x2 is the horizontal distance between the first optical center point d and the fourth target point 4, x4 is the horizontal distance between the first target point 1 and the first center point g, x7 is the horizontal distance between the fourth target point 4 and the fifth target point 5, D is the parallax of the telephoto camera at the third moment and the short - focus camera at the fourth moment with respect to the first spatial point a, and x8 is the horizontal distance between the fourth pixel point c and the second target point 2.

[0074] Please continue to refer to Figure 2 , the first spatial point a, the fourth pixel point c, and the second target point 2 form a triangle △ac2. The first spatial point a, the third target point 3, and the fifth target point 5 form a triangle △a35. The triangle △ac2 and the triangle △a35 are similar triangles.

[0075] Using the parallax principle and combining formulas (1) - (10), the first depth relationship can be determined:

[0076]

[0077] Z t1 = Z t0 - S (10)

[0078] Among them, t0 is the third moment, Z t0 is the depth of the first spatial point relative to the first optical center point of the short - focus camera at the third moment, t1 is the fourth moment, Z t1 is the depth of the first spatial point relative to the second optical center point of the short - focus camera at the fourth moment.

[0079] Based on the first spatial point, the third pixel point corresponding to the third imaging position, the fourth pixel point corresponding to the fourth imaging position, the first optical center point of the short - focus camera at the third moment, the second optical center point of the short - focus camera at the fourth moment, and the third optical center point of the long - focus camera at the third moment, the embodiment of the present application uses the principle of similar triangles and the principle of parallax to determine the first depth relationship formula, which can ensure the validity of the first depth relationship formula and can ensure the accuracy of determining the depth corresponding to the first pixel point by using the first depth relationship formula.

[0080] In an embodiment, the above - mentioned first depth relationship formula is formula (11):

[0081]

[0082] Wherein, t1 is the fourth moment, Z t1 is the depth of the first spatial point relative to the second optical center point of the short - focus camera at the fourth moment; B is the baseline length between the short - focus camera and the long - focus camera; f l is the focal length of the long - focus camera; f s is the focal length of the short - focus camera; S is the second moving distance of the stereoscopic imaging device from the third moment to the fourth moment; x1 is the horizontal distance between the third pixel point corresponding to the third imaging position and the second center point of the imaging plane of the long - focus camera at the third moment, and x3 is the horizontal distance between the fourth pixel point corresponding to the fourth imaging position and the first center point of the imaging plane of the short - focus camera at the fourth moment.

[0083] Optionally, the second moving distance of the stereoscopic imaging device from the third moment to the fourth moment is determined by formula (12):

[0084]

[0085] Wherein, n is the total number of vehicle speed acquisitions in the time period from t0 to t1, Δt is the time interval between adjacent vehicle speed acquisition moments, and v(ti) is the vehicle speed acquired at the i - th moment.

[0086] By using the above - mentioned first depth relationship formula, the embodiment of the present application can improve the efficiency and accuracy of determining the depth corresponding to the first pixel point. In addition, the vehicle speed collected from the first moment to the second moment can be substituted into formula (12), and the first moving distance of the stereoscopic imaging device from the first moment to the second moment can be accurately determined through integration, thereby further improving the accuracy of determining the depth corresponding to the first pixel point by using the above - mentioned first depth relationship formula.

[0087] In an embodiment, the outputting the stereoscopic image corresponding to the second moment according to the depth corresponding to each first pixel point in step S103 includes:

[0088] For each first pixel point, based on the principle of similar triangles, the spatial coordinates corresponding to the first pixel point are determined by using the depth corresponding to the first pixel point, the first imaging position corresponding to the first pixel point in the short-focus camera, and the focal length of the short-focus camera;

[0089] According to the spatial coordinates corresponding to each first pixel point, the target stereo image corresponding to the second moment is output.

[0090] Optionally, based on the principle of similar triangles and combined with formulas (11), (13), and (14), the spatial coordinates corresponding to the fourth pixel point are determined:

[0091]

[0092] wherein, the spatial coordinates corresponding to the fourth pixel point are (X t1 , Y t1 , Z t1 ), X t1 is the horizontal distance of the first spatial point relative to the imaging plane of the short-focus camera from the first center point at the fourth moment, Y t1 is the vertical distance of the first spatial point relative to the imaging plane of the short-focus camera from the first center point at the fourth moment, x3 is the horizontal distance between the fourth pixel point corresponding to the fourth imaging position and the first center point of the imaging plane of the short-focus camera at the fourth moment, and y3 is the vertical distance corresponding to the fourth imaging position. Optionally, the horizontal and vertical directions mentioned in the embodiments of the present application are two mutually perpendicular directions within the imaging plane of the short-focus camera or the long-focus camera.

[0093] Furthermore, the depth corresponding to the first pixel point, the horizontal and vertical distances corresponding to the first imaging position, and the focal length of the short-focus camera can be substituted into the above formulas (13) and (14) to obtain the spatial coordinates corresponding to the first pixel point.

[0094] Optionally, the target stereo image corresponding to the second moment can be the first stereo image corresponding to the first target area at the second moment. Specifically, based on the spatial targets corresponding to each first pixel point in the first target area, the spatial points whose imaging positions at the second moment are located in the first target area are restored according to a preset ratio to obtain the first stereo image corresponding to the first target area at the second moment, and the first stereo image is output as the target stereo image corresponding to the second moment. In combination with Figure 2 Regarding the description of the target stereo image, the target stereo image corresponding to the second moment may include the stereo image of area D1 and the stereo image of area D3.

[0095] In an embodiment of the present application, by using the principle of similar triangles, based on the depth corresponding to the first pixel point, the first imaging position corresponding to the first pixel point in the short-focus camera, and the focal length of the short-focus camera, the spatial coordinates corresponding to the first pixel point can be quickly and accurately determined. Further, based on the spatial coordinates corresponding to each first pixel point, the target stereoscopic image corresponding to the second moment is output, which can improve the efficiency and accuracy of stereoscopic imaging of the first target area.

[0096] In one embodiment, the step of outputting the target stereoscopic image corresponding to the second moment according to the depth corresponding to each first pixel point in step S103 includes:

[0097] Determine the first stereoscopic image corresponding to the first target area at the second moment according to the depth corresponding to each first pixel point;

[0098] Obtain the second target area; the second target area is the area in the second short-focus image that does not overlap with the first short-focus image and overlaps with the second long-focus image;

[0099] For each fifth pixel point in the second target area, according to the fifth imaging position corresponding to the fifth pixel point and the short-focus camera and the sixth imaging position corresponding to the sixth pixel point and the long-focus camera, use the second depth relationship formula to determine the depth corresponding to the fifth pixel point; the sixth pixel point is the pixel point in the second long-focus image that matches the fifth pixel point; the second depth relationship formula is determined based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth moment, the eighth imaging position of the second spatial point in the long-focus camera at the fifth moment, using the principle of similar triangles and the principle of parallax;

[0100] Determine the second stereoscopic image corresponding to the second target area at the second moment according to the depth corresponding to each fifth pixel point;

[0101] Output the target stereoscopic image corresponding to the second moment according to the first stereoscopic image and the second stereoscopic image.

[0102] Optionally, according to the above pixel matching result, the area in the second short-focus image that does not match the pixel points in the first short-focus image and matches the pixel points in the second long-focus image can be determined as the second target area, and the area in the second short-focus image corresponding to the sixth field of view area can also be determined as the second target area, where the sixth field of view area is the area in the above third field of view area that does not overlap with the above first field of view area and overlaps with the above fourth field of view area.

[0103] Optionally, the second depth relationship can be a relationship pre-stored in the stereoscopic imaging device or a relationship derived immediately. The fifth moment and the above-mentioned second moment can be the same moment or different moments. When the fifth moment and the second moment are different moments, the fifth moment is a historical moment earlier than the second moment. And when determining the depth corresponding to the fifth pixel point using the second depth relationship, the second depth relationship is a relationship pre-stored in the stereoscopic imaging device. It can be understood that in this case, the second depth relationship is a fixed formula and can be directly called when actually needed. When the fifth moment and the second moment are the same moment, when determining the depth corresponding to the fifth pixel point using the second depth relationship, the above-mentioned second depth relationship is a relationship immediately derived by the stereoscopic imaging device. It can be understood that in this case, the second depth relationship is a non-fixed formula and can be immediately derived when actually needed.

[0104] Optionally, the second depth relationship is a relationship between the horizontal distance corresponding to the seventh imaging position, the horizontal distance corresponding to the eighth imaging position, the baseline distance between the short-focus camera and the long-focus camera, and the focal lengths of the short-focus camera and the long-focus camera.

[0105] Furthermore, the horizontal distance corresponding to the fifth imaging position, the horizontal distance corresponding to the sixth imaging position, the baseline distance between the short-focus camera and the long-focus camera, and the focal lengths of the short-focus camera and the long-focus camera can be substituted into the second depth relationship to obtain the depth corresponding to the fifth pixel point.

[0106] Furthermore, using the principle of similar triangles, based on the depth corresponding to the fifth pixel point, in combination with formulas (13) and (14), the spatial target corresponding to the fifth pixel point can be determined. And based on the spatial targets corresponding to each fifth pixel point in the second target area, the spatial points where the imaging position at the second moment is located in the second target area can be restored according to a preset ratio to obtain the second stereoscopic image corresponding to the second target area at the second moment.

[0107] Optionally, the target stereoscopic image corresponding to the second moment can be a stereoscopic image obtained by splicing the first stereoscopic image corresponding to the first target area at the second moment and the second stereoscopic image corresponding to the second target area at the second moment. Combining Figure 3 Regarding the description of the target stereoscopic image, the target stereoscopic image corresponding to the second moment can be the splicing result of the stereoscopic images of area D1, area D2, and area D3.

[0108] In the embodiment of the present application, by determining the region in the second short - focus image that does not overlap with the first short - focus image and overlaps with the second long - focus image, and for each fifth pixel point in this region, based on the fifth imaging position corresponding to the short - focus camera of the fifth pixel point and the sixth imaging position corresponding to the long - focus camera of the sixth pixel point, using the second depth relationship formula, the depth corresponding to the fifth pixel point is determined, and based on the depth corresponding to the fifth pixel point, the second stereoscopic image corresponding to the second target region at the second moment is determined. Finally, by stitching the first stereoscopic image and the second stereoscopic image, the target stereoscopic image corresponding to the second moment is determined and output, so that not only can stereoscopic imaging be performed on the overlapping region between the short - focus image and the long - focus image collected at the same moment, but also stereoscopic imaging can be performed on the non - overlapping region between the short - focus image and the long - focus image collected at the same moment, thereby being able to expand the stereoscopic imaging range of cameras with different focal lengths.

[0109] In one embodiment, outputting the target stereoscopic image corresponding to the second moment according to the depth corresponding to each first pixel point in step S103 includes:

[0110] Determining the first stereoscopic image corresponding to the first target region at the second moment according to the depth corresponding to each first pixel point;

[0111] Obtaining a third target region; the third target region is the region in the first short - focus image that overlaps with the first long - focus image;

[0112] For each seventh pixel point in the third target region, according to the ninth imaging position corresponding to the short - focus camera of the seventh pixel point and the tenth imaging position corresponding to the long - focus camera of the eighth pixel point, using the second depth relationship formula, the depth corresponding to the seventh pixel point is determined; the eighth pixel point is the pixel point in the first long - focus image that matches the seventh pixel point; the second depth relationship formula is determined based on the seventh imaging position of the second spatial point in the short - focus camera at the fifth moment and the eighth imaging position of the second spatial point in the long - focus camera at the fifth moment, using the principle of similar triangles and the principle of parallax.

[0113] Determining the third stereoscopic image corresponding to the third target region at the first moment according to the depth corresponding to each seventh pixel point;

[0114] Outputting the target stereoscopic image corresponding to the second moment according to the first stereoscopic image and the third stereoscopic image.

[0115] Optionally, according to the above pixel matching result, the region where the pixel points in the first short - focus image match the pixel points in the first long - focus image can be determined as the third target region, or the region in the first short - focus image corresponding to the seventh field - of - view region can be determined as the third target region, where the seventh field - of - view region is the region in the above first field - of - view region that overlaps with the above second field - of - view region.

[0116] Optionally, the fifth moment and the above-mentioned first moment can be the same moment or different moments. When the fifth moment and the first moment are different moments, the fifth moment is a historical moment earlier than the first moment. And when determining the depth corresponding to the seventh pixel point using the second depth relationship formula, the second depth relationship formula is a formula pre-stored in the stereoscopic imaging device. It can be understood that in this case, the second depth relationship formula is a fixed formula and can be directly called when actually needed. When the fifth moment and the first moment are the same moment, when determining the depth corresponding to the seventh pixel point using the second depth relationship formula, the above-mentioned second depth relationship formula is a formula instantaneously derived by the stereoscopic imaging device. It can be understood that in this case, the second depth relationship formula is a non-fixed formula and can be instantaneously derived when actually needed.

[0117] Optionally, the horizontal distance corresponding to the ninth imaging position, the horizontal distance corresponding to the tenth imaging position, the baseline distance between the short-focus camera and the long-focus camera, and the focal lengths of the short-focus camera and the long-focus camera can be substituted into the second depth relationship formula to obtain the depth corresponding to the seventh pixel point.

[0118] Furthermore, using the principle of similar triangles, based on the depth corresponding to the seventh pixel point, in combination with formulas (13) and (14), the spatial target corresponding to the seventh pixel point can be determined. And based on the spatial targets corresponding to each seventh pixel point in the third target area, the spatial points of the imaging position at the first moment located in the third target area can be restored according to a preset ratio to obtain the third stereoscopic image corresponding to the third target area at the first moment.

[0119] Optionally, the target stereoscopic image corresponding to the above-mentioned second moment can be a stereoscopic image obtained by splicing the first stereoscopic image corresponding to the first target area at the second moment and the third stereoscopic image corresponding to the third target area at the first moment. Combining Figure 3 Regarding the target stereoscopic image, the target stereoscopic image corresponding to the second moment can be the splicing result of the stereoscopic images of area D1, area D3, and area A.

[0120] Optionally, the target stereoscopic image corresponding to the above-mentioned second moment can also be a stereoscopic image obtained by splicing the first stereoscopic image corresponding to the first target area at the second moment, the second stereoscopic image corresponding to the second target area at the second moment, and the third stereoscopic image corresponding to the third target area at the first moment. Combining Figure 3 Regarding the target stereoscopic image, the target stereoscopic image corresponding to the second moment can be the splicing result of the stereoscopic images of area D1, area D2, area D3, and area A.

[0121] Optionally, before stitching the third stereoscopic image corresponding to the first moment using the third target region to obtain the target stereoscopic image corresponding to the second moment, the third stereoscopic image can be cropped, and only the stereoscopic image corresponding to the space in front of the stereoscopic imaging device in the third stereoscopic image is used for stitching. In this way, only the stereoscopic image of the space in front of the stereoscopic imaging device can be presented in the target stereoscopic image corresponding to the second moment, improving the stereoscopic imaging visual effect.

[0122] In the embodiment of the present application, by determining the region overlapping between the first short - focus image and the first long - focus image, and for each seventh pixel point in this region, based on the ninth imaging position corresponding to the seventh pixel point and the short - focus camera and the tenth imaging position corresponding to the eighth pixel point and the long - focus camera, using the second depth relationship formula to determine the depth corresponding to the seventh pixel point, and based on the depth corresponding to the seventh pixel point, determining the third stereoscopic image corresponding to the third target region at the first moment. Finally, by stitching the first stereoscopic image and the third stereoscopic image, the target stereoscopic image corresponding to the second moment is determined and output, enabling the simultaneous presentation of the stereoscopic image corresponding to the overlapping region in the short - focus image and the long - focus image collected at the previous same moment, and the stereoscopic image corresponding to the non - overlapping region in the short - focus image and the long - focus image collected at the next same moment, thereby expanding the stereoscopic imaging range of cameras with different focal lengths.

[0123] In an embodiment, before determining the depth corresponding to the fifth pixel point according to the fifth imaging position corresponding to the fifth pixel point and the short - focus camera and the sixth imaging position corresponding to the sixth pixel point and the long - focus camera using the second depth relationship formula, or before determining the depth corresponding to the seventh pixel point according to the ninth imaging position corresponding to the seventh pixel point and the short - focus camera and the tenth imaging position corresponding to the eighth pixel point and the long - focus camera using the second depth relationship formula, the stereoscopic imaging method provided by the embodiment of the present application further includes:

[0124] Obtain the second spatial point, the ninth pixel point corresponding to the seventh imaging position, the tenth pixel point corresponding to the eighth imaging position, the fourth optical center point of the short - focus camera at the fifth moment, and the fifth optical center point of the long - focus camera at the fifth moment;

[0125] Determine the intersection point of the imaging plane of the short - focus camera at the fifth moment and the seventh straight line as the eighth target point; the seventh straight line is a straight line passing through the fourth optical center point and parallel to the eighth straight line; the eighth straight line is a straight line passing through the second spatial point and the fifth optical center point.

[0126] Determine the intersection point of the ninth straight line and the eighth straight line as the ninth target point; the ninth straight line is a straight line passing through the fourth optical center point and parallel to the imaging plane of the short - focus camera at the fifth moment.

[0127] Using the principle of similar triangles, based on the tenth pixel point, the ninth target point, the fourth optical center point, and the fifth optical center point, determine the third distance relationship; the third distance relationship is used to solve the distance between the fourth optical center point and the ninth target point;

[0128] Using the principle of similar triangles, based on the eighth target point, the ninth pixel point, the tenth pixel point, the fourth optical center point, the fifth optical center point, and the third distance relationship, determine the fourth distance relationship; the fourth distance relationship is used to solve the distance between the tenth pixel point and the ninth pixel point;

[0129] Using the parallax principle, based on the third distance relationship and the fourth distance relationship, determine the second depth relationship.

[0130] Please refer to Figure 3 , at the fifth moment, the Field Of View (FOV) area of the short - focus camera is the 5# sector area or the 6# sector area, and the FOV area of the long - focus camera is the 7# sector area or the 8# sector area. From Figure 3 it can be seen that there is an overlapping area D2 or area A in the FOV area of the short - focus camera at the fifth moment with the FOV area of the long - focus camera at the fifth moment. Correspondingly, the image area corresponding to area D2 or area A in the image collected by the short - focus camera at the fifth moment overlaps with the image collected by the long - focus camera at the fourth moment.

[0131] Please continue to refer to Figure 3 , the second target space point a' can be any space point in area D2 or area A. The intersection point t of the ninth straight line and the tenth straight line, the ninth target point 9, and the fifth optical center point s form a triangle △ts9. The fifth optical center point s, the tenth pixel point q, and the fourth center point u of the imaging plane of the long - focus camera at the fifth moment form a triangle △squ. The triangle △ts9 and the triangle △squ are similar triangles. Among them, the tenth straight line is the straight line passing through the fifth optical center point s and the fourth center point u.

[0132] The principle of similar triangles can be used, combined with formulas (15) and (16), to determine the third distance relationship:

[0133]

[0134] A' = B - x'5 (16)

[0135] Among them, x′1 is the horizontal distance between the tenth pixel point q and the fourth center point u, that is, the horizontal distance corresponding to the eighth imaging position, x′5 is the horizontal distance between the ninth target point 9 and the intersection point t, and A′ is the horizontal distance between the fourth optical center point r and the ninth target point 9.

[0136] Please continue to refer to Figure 3, at the fifth moment, the imaging plane of the short - focus camera forms a triangle △rv8 with the fifth center point v, the eighth target point 8, and the fourth optical center point r. The triangle △rv8 and the triangle △squ are similar triangles. The eighth target point 8 and the tenth pixel point q form a line segment Lq8, and the fourth optical center point r and the ninth target point 9 form a line segment Lr9. The length of the line segment Lq8 is equal to the length of the line segment Lr9.

[0137] The principle of similar triangles can be used, combined with formulas (15) - (19), to determine the fourth distance relationship:

[0138]

[0139] D' = x′2 + x'3 (18)

[0140] x′4 = A' - D′ (19)

[0141] Among them, x′3 is the horizontal distance between the eighth target point 8 and the fifth center point v, x′2 is the horizontal distance between the ninth pixel point p and the fifth center point v, that is, the horizontal distance corresponding to the seventh imaging position, D’ is the parallax of the tele - focus camera and the short - focus camera for the second spatial point a’ at the fifth moment, and x′4 is the horizontal distance between the ninth pixel point p and the tenth pixel point q.

[0142] Please continue to refer to Figure 3 , the second spatial point a’, the ninth pixel point p, and the tenth pixel point q form a triangle △a’pq, and the second spatial point a’, the fourth optical center point r, and the ninth target point 9 form a triangle △a’r9. The triangle △a’pq and the triangle △a’r9 are similar triangles.

[0143] The parallax principle can be used, combined with formulas (15) - (20), to determine the second depth relationship (21):

[0144]

[0145] Among them, t2 is the fifth moment, Z t2 is the depth of the second spatial point relative to the fourth optical center point of the short - focus camera at the fifth moment.

[0146] Based on the second spatial point, the ninth pixel point corresponding to the seventh imaging position, the tenth pixel point corresponding to the eighth imaging position, the fourth optical center point of the short - focus camera at the fifth moment, and the fifth optical center point of the tele - focus camera at the fifth moment, the embodiments of the present application use the principle of similar triangles and the parallax principle to determine the second depth relationship, which can ensure the effectiveness of the second depth relationship and can ensure the accuracy of determining the depth corresponding to the fifth pixel point or the seventh pixel point using the second depth relationship.

[0147] Please refer to Figure 4, a stereoscopic imaging method provided by an embodiment of the present application may include the following steps:

[0148] Step S201: Calibrate the internal and external parameters of the short-focus camera and the long-focus camera to obtain the focal length of the short-focus camera, the focal length of the long-focus camera, and the baseline distance between the short-focus camera and the long-focus camera;

[0149] Step S202: Obtain the first short-focus image and the second short-focus image collected by the short-focus camera at the first moment and the second moment respectively, and the first long-focus image and the second long-focus image collected by the long-focus camera at the first moment and the second moment respectively;

[0150] Step S203: Match the pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image to obtain a pixel matching result;

[0151] Step S204: Determine the target area according to the pixel matching result; the target area includes at least one of the first target area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image and overlaps with the first long-focus image, the second target area in the second short-focus image that does not overlap with the first short-focus image and overlaps with the second long-focus image, and the third target area in the first short-focus image that overlaps with the first long-focus image;

[0152] Step S205: For each pixel point in the target area, use the corresponding depth relationship formula to determine the depth corresponding to each pixel point in the target area;

[0153] Step S206: Use the principle of similar triangles to determine the spatial coordinates corresponding to each pixel point in the target area based on the depth corresponding to each pixel point in the target area;

[0154] Step S207: Output the target stereoscopic image corresponding to the second moment based on the spatial coordinates corresponding to each pixel point in the target area.

[0155] It is worth mentioning that during the process of matching the pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image in step S203, if any of the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image has been cropped or scaled in size, then the focal length of the short-focus camera, the focal length of the long-focus camera, and the baseline distance between the short-focus camera and the long-focus camera calibrated in step S201, the position of the center point of the imaging plane of the short-focus camera, and the position of the center point of the imaging plane of the long-focus camera need to be updated so as to accurately determine the depth corresponding to each pixel point in the target area.

[0156] In addition, the specific implementation processes of steps S203 to S206 refer to the above-mentioned embodiments and will not be elaborated here.

[0157] Through the above steps S201 to S207, the embodiment of the present application can output the stereoscopic image of the first target area as the target stereoscopic image at the second moment, or output the spliced image of the stereoscopic image of the first target area and the stereoscopic image of the second target area as the target stereoscopic image at the second moment, or output the spliced image of the stereoscopic image of the first target area and the stereoscopic image of the third target area as the target stereoscopic image at the second moment, or output the spliced image of the stereoscopic image of the first target area, the stereoscopic image of the second target area and the stereoscopic image of the third target area as the target stereoscopic image at the second moment, which can expand the range and flexibility of stereoscopic imaging of cameras with different focal lengths.

[0158] It is worth mentioning that in the case where the stereoscopic imaging device is a vehicle, the vehicle can detect distant targets through a telephoto camera, which can meet the usage requirements of the intelligent driving function. It can also expand the stereoscopic imaging range of cameras with different focal lengths by implementing the stereoscopic imaging method provided by the embodiment of the present application, which can meet the usage requirements of the chassis magic carpet function. Furthermore, it can balance the intelligent driving function and the chassis magic carpet function and improve the usage effects of the intelligent driving function and the chassis magic carpet function.

[0159] Please refer to Figure 5 , the embodiment of the present application also provides a stereoscopic imaging device 500, and the stereoscopic imaging device 500 includes a first acquisition module 501, a second acquisition module 502, a depth determination module 503 and a stereoscopic imaging module 505.

[0160] Among them, the first acquisition module 501 is used to acquire a first short-focus image and a second short-focus image acquired by the short-focus camera at the first moment and the second moment respectively, and a first telephoto image and a second telephoto image acquired by the telephoto camera at the first moment and the second moment respectively; where the first moment is the previous image acquisition moment of the second moment;

[0161] The second acquisition module 502 is used to acquire a first target area; where the first target area is the area in the second short-focus image that does not overlap with the first short-focus image and the second telephoto image, and overlaps with the first telephoto image;

[0162] The depth determination module 503 is configured to, for each first pixel point in the first target area, determine the depth corresponding to the first pixel point by using a first depth relationship formula based on the first moving distance of the stereo imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera; the second pixel point is the pixel point in the first long-focus image that matches the first pixel point; wherein, the first depth relationship formula is determined based on the third imaging position of the first spatial point in the long-focus camera at the third moment and the fourth imaging position of the first spatial point in the short-focus camera at the fourth moment by using the principle of similar triangles and the principle of parallax; the third moment is the previous image acquisition moment of the fourth moment.

[0163] The stereo imaging module 505 is configured to output a target stereo image corresponding to the second moment according to the depth corresponding to each first pixel point.

[0164] The stereo imaging device provided by the embodiments of the present application can implement each step of the above-mentioned stereo imaging method embodiments and achieve the same technical effects. To avoid repetition, details are not described here again.

[0165] The embodiments of the present application further provide a stereo imaging device, including a processor and a memory. A program or instruction that can run on the processor is stored on the memory. When the program or instruction is executed by the processor, each step of the above-mentioned stereo imaging method embodiments is implemented and the same technical effects can be achieved. To avoid repetition, details are not described here again.

[0166] Figure 6 The following is a schematic hardware structure diagram of the stereo imaging device for implementing the embodiments of the present application. The stereo imaging device includes:

[0167] A processor 601, which can be implemented by using a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0168] A memory 602, which can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 602 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 602 and are called by the processor 601 to execute the stereo imaging method of the embodiments of the present application;

[0169] An input / output interface 603 for implementing information input and output;

[0170] A communication interface 604 for implementing communication interaction between this device and other devices, which can implement communication through a wired manner (such as USB, network cable, etc.) or through a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0171] A bus 605 for transmitting information between various components of the device (such as a processor 601, a memory 602, an input / output interface 603, and a communication interface 604);

[0172] Among them, the processor 601, the memory 602, the input / output interface 603, and the communication interface 604 achieve communication connections with each other inside the device through the bus 605.

[0173] The three-dimensional imaging device provided by the embodiment of the present application can implement each step of the above-mentioned three-dimensional imaging method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0174] The embodiment of the present application also provides a computer-readable storage medium. A program or instruction is stored on the computer-readable storage medium. When the program or instruction is executed by a processor, each step of the above-mentioned three-dimensional imaging method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be elaborated here.

[0175] Among them, the processor is the processor in the three-dimensional imaging device described in the above embodiment. The computer-readable storage medium includes computer-readable storage media such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk, or an optical disc, etc.

[0176] The embodiment of the present application further provides a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run a program or instruction to implement each step of the above-mentioned three-dimensional imaging method embodiment, and the same technical effect can be achieved. To avoid repetition, it will not be elaborated here.

[0177] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-on-chip, a system chip, a chip system, or a system-on-chip, etc.

[0178] The embodiment of the present application provides a computer program product. The program product is stored in a storage medium. The program product is executed by at least one processor to implement each step of the above-mentioned three-dimensional imaging method embodiment, and the same technical effect can be achieved. To avoid repetition, it will not be elaborated here.

[0179] It should be noted that in this text, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0180] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0181] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Those of ordinary skill in the art, under the inspiration of the present application and without departing from the purpose of the present application and the scope protected by the claims, can still make many forms, all of which fall within the protection scope of the present application.

Claims

1. A stereoscopic imaging method, characterized in that: Applied to a stereoscopic imaging device, the stereoscopic imaging device includes a short-focus camera and a long-focus camera, and the stereoscopic imaging method includes: Acquire a first short-focus image and a second short-focus image captured by the short-focus camera at a first moment and a second moment, respectively, and a first long-focus image and a second long-focus image captured by the long-focus camera at the first moment and the second moment, respectively; wherein the first moment is a previous image capture moment of the second moment; Acquire a first target area; wherein the first target area is an area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image, and overlaps with the first long-focus image; For each first pixel point in the first target area, based on the first moving distance of the stereoscopic imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the telephoto camera, a first depth relationship formula is used to determine the depth corresponding to the first pixel point; the second pixel point is a pixel point in the first telephoto image that matches the first pixel point; wherein the first depth relationship formula is determined based on the third imaging position of the first spatial point in the telephoto camera at the third moment and the fourth imaging position of the first spatial point in the short-focus camera at the fourth moment, using the similar triangle principle and the parallax principle; the third moment is the previous image acquisition moment of the fourth moment; According to the depth corresponding to each of the first pixel points, a target stereoscopic image corresponding to the second moment is output.

2. The stereoscopic imaging method according to claim 1, characterized in that: Before determining the depth corresponding to the first pixel point by using a first depth relationship equation based on the first moving distance of the stereoscopic imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera, the stereoscopic imaging method further includes: Acquire the first spatial point, a third pixel point corresponding to the third imaging position, a fourth pixel point corresponding to the fourth imaging position, a first optical center point of the short-focus camera at the third moment, a second optical center point of the short-focus camera at the fourth moment, and a third optical center point of the long-focus camera at the third moment; Determine the intersection of the imaging plane of the short-focus camera at the fourth moment and the first straight line as the first target point; the first straight line is a straight line passing through the second optical center point and parallel to the second straight line; the second straight line is a straight line passing through the first spatial point and the third optical center point; Determine the intersection of a third straight line and the second straight line as the second target point; the third straight line is a straight line passing through the fourth pixel point and parallel to the imaging plane of the short-focus camera at the fourth moment; The intersection of the fourth straight line and the fifth straight line is determined as the third target point; the fourth straight line is a straight line passing through the first optical center point and parallel to the imaging plane of the short-focus camera at the third moment; the fifth straight line is a straight line passing through the first spatial point and the second optical center point; determining an intersection point of the first straight line and the fourth straight line as a fourth target point; determining an intersection point of the fourth straight line and the second straight line as a fifth target point; Using the similar triangle principle, a first distance relationship is determined based on the third pixel point, the fourth pixel point, the third target point, the fifth target point, the first optical center point, the second optical center point and the third optical center point; the first distance relationship is used to solve the distance between the third target point and the fifth target point; Using the similar triangle principle, a second distance relationship is determined based on the third pixel point, the fourth pixel point, the first optical center point, the second optical center point, the third optical center point, the first target point, the fourth target point and the fifth target point; the second distance relationship is used to solve the distance between the fourth pixel point and the second target point; The first depth relationship equation is determined based on the first distance relationship equation and the second distance relationship equation by utilizing the parallax principle.

3. The stereoscopic imaging method according to claim 1, characterized in that: The first depth relationship is: Wherein, t1 is the fourth moment, Z t1 is the depth of the first spatial point relative to the second optical center point of the short-focus camera at the fourth moment; B is the baseline length between the short-focus camera and the long-focus camera; f l f is the focal length of the telephoto camera; s is the focal length of the short-focus camera; S is the second moving distance of the stereoscopic imaging device from the third moment to the fourth moment; x1 is the lateral distance between the third pixel point corresponding to the third imaging position and the second center point of the imaging plane of the telephoto camera at the third moment, and x3 is the lateral distance between the fourth pixel point corresponding to the fourth imaging position and the first center point of the imaging plane of the short-focus camera at the fourth moment.

4. The stereoscopic imaging method according to claim 1, wherein: The step of outputting a target stereoscopic image corresponding to the second moment according to the depth corresponding to each of the first pixel points includes: Determine, according to the depth corresponding to each of the first pixel points, a first stereoscopic image corresponding to the first target area at the second moment; Acquire a second target area; the second target area is an area in the second short-focus image that does not overlap with the first short-focus image and overlaps with the second long-focus image; For each fifth pixel in the second target area, according to a fifth imaging position corresponding to the fifth pixel and the short-focus camera and a sixth imaging position corresponding to the sixth pixel and the telephoto camera, a second depth relationship formula is used to determine the depth corresponding to the fifth pixel; the sixth pixel is a pixel in the second telephoto image that matches the fifth pixel; the second depth relationship formula is based on a seventh imaging position of the second spatial point in the short-focus camera at a fifth moment and an eighth imaging position of the second spatial point in the telephoto camera at the fifth moment, and is determined by using the similar triangle principle and the parallax principle; Determine, according to the depth corresponding to each of the fifth pixels, a second stereoscopic image corresponding to the second target area at the second moment; The target stereoscopic image is output according to the first stereoscopic image and the second stereoscopic image.

5. The stereoscopic imaging method according to claim 1, characterized in that: The step of outputting a target stereoscopic image corresponding to the second moment according to the depth corresponding to each of the first pixel points includes: Determine, according to the depth corresponding to each of the first pixel points, a first stereoscopic image corresponding to the first target area at the second moment; Acquire a third target area; the third target area is an area in the first short-focus image that overlaps with the first long-focus image; For each seventh pixel in the third target area, according to the ninth imaging position corresponding to the seventh pixel and the short-focus camera and the tenth imaging position corresponding to the eighth pixel and the telephoto camera, the depth corresponding to the seventh pixel is determined by using a second depth relationship formula; the eighth pixel is a pixel in the first telephoto image that matches the seventh pixel; the second depth relationship formula is determined based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth moment and the eighth imaging position of the second spatial point in the telephoto camera at the fifth moment, by using the similar triangle principle and the parallax principle; Determine, according to the depth corresponding to each of the seventh pixel points, a third stereoscopic image corresponding to the third target area at the first moment; The target stereoscopic image is output according to the first stereoscopic image and the third stereoscopic image.

6. The stereoscopic imaging method according to claim 1, characterized in that: The step of outputting a stereoscopic image corresponding to the second moment according to the depth corresponding to each of the first pixel points includes: For each of the first pixel points, using the similar triangle principle, based on the depth corresponding to the first pixel point, the first imaging position corresponding to the first pixel point and the short-focus camera, and the focal length of the short-focus camera, determine the spatial coordinates corresponding to the first pixel point; The target stereoscopic image is output according to the spatial coordinates corresponding to each of the first pixel points.

7. The stereoscopic imaging method according to claim 1, characterized in that: The obtaining of the first target area comprises: Respectively acquiring spatial target categories corresponding to pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image; Using a pixel matching algorithm, according to the spatial target category corresponding to the pixel points, pixel points in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image are matched to obtain a pixel matching result; The first target area is determined according to the pixel matching result.

8. A stereoscopic imaging device, characterized in that: The stereoscopic imaging device comprises a first acquisition module, a second acquisition module, a depth determination module and a stereoscopic imaging module; The first acquisition module is used to acquire a first short-focus image and a second short-focus image acquired by the short-focus camera at a first moment and a second moment, respectively, and a first long-focus image and a second long-focus image acquired by the long-focus camera at the first moment and the second moment, respectively; wherein the first moment is the image acquisition moment before the second moment; The second acquisition module is used to acquire a first target area; wherein the first target area is an area in the second short-focus image that does not overlap with the first short-focus image and the second long-focus image, and overlaps with the first long-focus image; The depth determination module is used to determine, for each first pixel in the first target area, the depth corresponding to the first pixel using a first depth relationship equation based on a first moving distance of the stereoscopic imaging device from the first moment to the second moment, a first imaging position corresponding to the first pixel in the short-focus camera, and a second imaging position corresponding to the second pixel in the telephoto camera; the second pixel is a pixel matching the first pixel in the first telephoto image; wherein the first depth relationship equation is determined based on a third imaging position of the first spatial point in the telephoto camera at a third moment and a fourth imaging position of the first spatial point in the short-focus camera at a fourth moment, using the principle of similar triangles and the principle of parallax; the third moment is the last image acquisition moment of the fourth moment; The stereo imaging module is used to output a target stereo image corresponding to the second moment according to the depth corresponding to each of the first pixel points.

9. A stereoscopic imaging device, characterized in that: The stereoscopic imaging device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the stereoscopic imaging method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the stereoscopic imaging method according to any one of claims 1 to 7 is implemented.

Images