Preview aiming method, binocular camera, preview aiming system, automobile, controller and medium

Abstract

The present invention provides a preview method, a binocular camera, a preview system, a car, a controller and a medium, wherein the method comprises extracting an area of ​​interest and performing polar coordinate conversion on a first road scan image and a second road scan image acquired based on a binocular camera in sequence to obtain a first polar coordinate area image and a second polar coordinate area image, obtaining a feature point matching set according to the registration result of the first polar coordinate area image and the second polar coordinate area image, obtaining a road area spatial position set according to the pixel position coordinates corresponding to each matching feature point in the feature point matching set, and obtaining a road surface three-dimensional reconstruction result according to a road surface preview area information set obtained based on the road surface spatial position set. The present invention performs binocular image registration based on the road surface matching constraint of the same optical axis, ensures that the registration process is simple and efficient, and improves the registration accuracy and registration effect, providing a reliable basis for road surface preview reconstruction and intelligent driving strategy formulation.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle vision system design, and in particular to a previewing method, a binocular camera, a previewing system, a vehicle, a controller and a medium. Background Art

[0002] With the continuous promotion and application of intelligent driving technology, the requirements for vehicle vision systems are gradually increasing. How to perform efficient and accurate preview detection of drivable road surfaces has gradually become a hot research direction in the field of intelligent driving.

[0003] Existing road surface preview systems for drivable roads are all based on binocular stereo vision technology. They use binocular cameras to detect road surfaces and identify the flatness of drivable roads based on the detection results, providing guidance for intelligent driving strategies. However, in actual applications, this technology requires multiple steps such as calibration, stereo correction, and registration (parallax estimation) for spatial recovery. Not only is the amount of calculation too large, but also because the effective field of view of the camera (overlapping field of view area) is easily affected by the baseline distance, the calibration accuracy of the camera is very high, and there are great application limitations. Summary of the invention

[0004] In order to solve the problems of the prior art, it is necessary to provide a preview method, a binocular camera, a preview system, a car, a controller and a medium in response to the above technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a road surface preview method, the method comprising the following steps:

[0006] Acquire a first road surface scanning image and a second road surface scanning image through a binocular camera; the first road surface scanning image and the second road surface scanning image are co-optical axis images;

[0007] Obtain a first region of interest image based on the first road surface scan image, and obtain a second region of interest image based on the second road surface scan image;

[0008] By polar coordinate conversion, a first polar coordinate region image is obtained based on the first region of interest image, and a second polar coordinate region image is obtained based on the second region of interest image;

[0009] Acquire a registration result of the first polar coordinate region image and the second polar coordinate region image, and obtain a feature point matching set according to the registration result;

[0010] According to the pixel position coordinates of each matching feature point pair in the feature point matching set, a road surface area spatial position set is obtained;

[0011] Obtaining a road surface preview area information set based on the road surface area spatial position set by converting the vehicle body coordinate system;

[0012] A three-dimensional reconstruction result of the road surface is obtained according to the road surface preview area information set.

[0013] In a second aspect, an embodiment of the present invention provides a binocular camera, the binocular camera comprising a first camera body, a second camera body, a first lens, a second lens, a semi-transparent and semi-reflective mirror, and a reflector;

[0014] The semi-transparent and semi-reflective mirror has a light-transmitting end aligned with the light-incident end of the reflector and a light-reflecting end aligned with the light-incident end of the second lens, and the semi-transparent and semi-reflective mirror is used to reflect and transmit the received incident light to obtain corresponding first reflected light and transmitted light;

[0015] The second lens has a light output end aligned with a light input end of the second camera body, and the second lens is used to converge the received first reflected light and emit it into the second camera body;

[0016] The reflector has a light output end aligned with the light input end of the first lens, and is used to reflect the received transmitted light to obtain a second reflected light parallel to the first reflected light;

[0017] The light emitting end of the first lens is aligned with the light incident end of the first camera body, and the first lens is used to converge the received second reflected light and emit it into the first camera body.

[0018] In a third aspect, an embodiment of the present invention provides a road preview system, the system comprising an image acquisition module, a region extraction module, a polar coordinate conversion module, a stereo registration module, a spatial information acquisition module, a preview information acquisition module and a three-dimensional reconstruction module;

[0019] The image acquisition module is used to obtain a first road surface scanning image and a second road surface scanning image through a binocular camera; the first road surface scanning image and the second road surface scanning image are co-optical axis images;

[0020] The region extraction module is used to obtain a first region of interest image based on the first road surface scanning image, and to obtain a second region of interest image based on the second road surface scanning image;

[0021] The polar coordinate conversion module is used to obtain a first polar coordinate region image based on the first region of interest image and obtain a second polar coordinate region image based on the second region of interest image through polar coordinate conversion;

[0022] The stereo registration module is used to obtain the registration result of the first polar coordinate area image and the second polar coordinate area image, and obtain a feature point matching set according to the registration result;

[0023] The spatial information acquisition module is used to obtain a road surface area spatial position set according to the pixel position coordinates of each matching feature point pair in the feature point matching set;

[0024] The preview information acquisition module is used to obtain a road surface preview area information set based on the road surface area spatial position set through vehicle body coordinate system conversion;

[0025] The three-dimensional reconstruction module is used to obtain a three-dimensional reconstruction result of the road surface according to the road surface preview area information set.

[0026] In a fourth aspect, an embodiment of the present invention further provides a car on which a computer device is deployed, wherein the computer device includes: a memory, a processor, and a road surface preview program stored in the memory and executable on the processor, wherein the road surface preview program is configured with steps for implementing the above-mentioned road surface preview method.

[0027] In the fifth aspect, an embodiment of the present invention further provides a controller, characterized in that the controller includes a processor and a memory, the processor and the memory are coupled, the memory is used for computer programs or instructions, and the processor is used to execute the computer programs or instructions in the memory, so that the controller performs the steps of the above-mentioned road surface preview method.

[0028] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0029] Compared with the prior art, on the one hand, the road preview method of the embodiment of the present invention performs binocular road stereogram registration by using the road matching constraint based on the same optical axis, which can not only avoid excessive calculation operations and ensure the simplicity and efficiency of the registration process, but also effectively improve the registration accuracy and registration effect, provide a reliable basis for road three-dimensional reconstruction, and further provide a reliable guidance basis for the formulation of intelligent driving strategies. On the other hand, the binocular camera of the embodiment of the present invention realizes the common path of the optical axis of binocular cameras of different focal lengths through the combined design of semi-transparent and semi-reflective and reflective mirrors, which can not only increase the camera's field of view, but also effectively avoid the camera's field of view being affected by the baseline distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic flow chart of a road surface preview method in an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of the field of view angle of the binocular camera in an embodiment of the present invention;

[0032] Figure 3is a schematic diagram of a road surface region of interest in an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of converting a road surface scan image into a road surface bird's-eye view in an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of a first region of interest image and a second region of interest image in an embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of the dual-focus imaging principle in an embodiment of the present invention;

[0036] Figure 7 is a schematic diagram of polar coordinate transformation in an embodiment of the present invention;

[0037] Figure 8 is a schematic diagram of binocular image registration in a Cartesian coordinate system and a polar coordinate system in an embodiment of the present invention;

[0038] Fig. 9 is a schematic diagram of feature point matching for binocular image registration in an embodiment of the present invention;

[0039] Fig.10 is a schematic diagram of the structure of a binocular camera in an embodiment of the present invention;

[0040] Fig.11 is another structural schematic diagram of a binocular camera in an embodiment of the present invention;

[0041] Fig.12 Schematic diagram of the layout of the vehicle-mounted binocular camera in an embodiment of the present invention;

[0042] Fig.13 It is a structural schematic diagram of a road surface preview system in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and beneficial effects of the present application clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the embodiments described below are part of the embodiments of the present invention and are only used to illustrate the present invention, but are not used to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In some embodiments, Figure 1 As shown, a road surface preview method is provided, the method comprising the following steps:

[0045] S11, acquiring a first road surface scanning image and a second road surface scanning image through a binocular camera; the first road surface scanning image and the second road surface scanning image are co-optical axis images;

[0046] In some embodiments, the first road scan image and the second road scan image can be understood as imaging effects at two different focal lengths acquired by using a binocular camera with an optical axis passing through the center of the camera. If the focal length of one camera in the binocular camera is Smaller, larger field of view (FOV), corresponding to a larger field of view, and the focal length of the other camera Larger, smaller field of view, correspondingly smaller field of view, then we can get Figure 2 The first road scan image and the second road scan image shown in FIG. and The size relationship is only an exemplary description. In actual applications, the size relationship of the focal lengths of the two cameras in the binocular camera can be adjusted or Larger, In smaller cases, it is sufficient that the focal lengths of the two cameras are different.

[0047] In some embodiments, a binocular camera is further configured to include a first camera body, a second camera body, a first lens, a second lens, a semi-transparent and semi-reflective mirror, and a reflector;

[0048] The semi-transparent and semi-reflective mirror has a light-transmitting end aligned with the light-incident end of the reflector and a light-reflecting end aligned with the light-incident end of the second lens, and the semi-transparent and semi-reflective mirror is used to reflect and transmit the received incident light to obtain corresponding first reflected light and transmitted light;

[0049] The second lens has a light output end aligned with a light input end of the second camera body, and the second lens is used to converge the received first reflected light and emit it into the second camera body;

[0050] The reflector has a light output end aligned with the light input end of the first lens, and is used to reflect the received transmitted light to obtain a second reflected light parallel to the first reflected light;

[0051] The light emitting end of the first lens is aligned with the light incident end of the first camera body, and the first lens is used to converge the received second reflected light and emit it into the first camera body.

[0052] The binocular camera realizes the common path of the optical axes of binocular cameras of different focal lengths through the combination design of semi-transparent and semi-reflective and reflective mirrors, which not only increases the camera's field of view, but also effectively avoids the camera's field of view being affected by the baseline distance. It can also provide binocular road scanning images with the same optical axis, and provide reliable support for simple, efficient and accurate binocular image stereo registration based on the road matching constraints of the same optical axis, thereby effectively improving the efficiency and accuracy of road 3D reconstruction.

[0053] S12. Obtain a first region of interest image based on the first road surface scanning image, and obtain a second region of interest image based on the second road surface scanning image; wherein the first region of interest image and the second region of interest image can be understood as rectangular areas intercepted from the first road surface scanning image and the second road surface scanning image according to an image area within a certain distance range in front of the vehicle that needs to be paid attention to in actual applications, also known as road surface ROI (Region of Interest) areas.

[0054] Considering the positional relationship between the binocular camera and the road surface in practical applications, the ROI area in the first road surface scanning image and the second road surface scanning image collected is as follows: Figure 3 The red area in the middle is actually presented as a trapezoid, and the interception and processing of the trapezoidal area is relatively cumbersome and requires a large amount of computing resources. In order to reduce the amount of calculation for subsequent image processing and ensure the efficiency and reliability of the extraction and analysis of the road area of ​​interest, in some embodiments, it is further configured to first convert the acquired road scan image into a bird's-eye view so that the required road area of ​​interest becomes a rectangular area before extracting the area of ​​interest image; specifically, the steps of obtaining the first area of ​​interest image based on the first road scan image and obtaining the second area of ​​interest image based on the second road scan image include:

[0055] By using the affine transformation principle, a first road surface bird's-eye view is obtained based on the first road surface scan image, and a second road surface bird's-eye view is obtained based on the second road surface scan image; wherein the first road surface bird's-eye view and the second road surface bird's-eye view can be understood as images obtained by performing view conversion on the first road surface scan image and the second road surface scan image, and the corresponding affine transformation formula is as follows:

[0056]

[0057] in, Represents the pixel coordinates in the bird's-eye view of the road surface; Represents the pixel coordinates of the road scan image; , , and The corresponding axis rotation and scaling parameters; and is the coordinate axis translation parameter; it should be noted that the parameter value in the specific affine transformation formula can be set according to actual needs and is not specifically limited here.

[0058] Through the above affine transformation, Figure 4 The first road surface bird's-eye view and the second road surface bird's-eye view shown in Figure (a) are converted into Figure 4The first road surface aerial view and the second road surface aerial view shown in Figure (b) are about to Figure 4 The four vertices of the trapezoid shown in Figure (a) are mapped to Figure 4 At the four vertices of the rectangle in the middle (b) figure, the camera perspective is changed to a bird's-eye view perspective to ensure that the road surface ROI area in the first road surface bird's-eye view and the second road surface bird's-eye view is a rectangular area. Since the camera is facing the road surface in the bird's-eye view perspective, it is convenient to extract the road surface ROI area efficiently and accurately, and it is also convenient to better estimate the depth information of the road surface ROI area. It can effectively solve the problem that the trapezoidal area is cumbersome to represent and calculate using polar coordinates, which is not conducive to improving the efficiency of subsequent image processing, and provide reliable guarantee for the efficiency of subsequent binocular image stereo registration.

[0059] According to the preset region of interest identification parameters, a first region of interest image is obtained based on the first road surface bird's-eye view, and a second region of interest image is obtained based on the second road surface bird's-eye view; wherein the preset region of interest identification parameters can be understood as regional position parameter information for identifying and intercepting the region of interest image from the road surface bird's-eye view, and in principle can be selected and set according to actual application requirements. In order to ensure the reliability of the region of interest interception in actual applications, in some embodiments, the preset region of interest identification parameters are further obtained by pre-calibration for subsequent extraction of the region of interest from the road surface scanning image collected in real time; specifically, the step of obtaining the preset region of interest identification parameters includes:

[0060] A first image to be annotated and a second image to be annotated including preset area identification points are obtained; wherein the preset area identification points can be understood as identification points arranged in the road area within the camera field of view for identifying the area of ​​interest of the road surface in front of the vehicle to be analyzed; in some embodiments, the identification range corresponding to the preset area identification points is further set to be 60 meters long and 2 meters wide in front of the vehicle ( ) area, that is, the area on the road in front of the vehicle After the marking points are placed in the area, a binocular camera is used to capture a first image to be marked and a second image to be marked that include the preset area marking points.

[0061] According to the preset area identification points, a first annotated image is obtained based on the first image to be annotated, and a second annotated image is obtained based on the second image to be annotated; wherein the first annotated image and the second annotated image can be understood as images obtained by annotating corresponding image position points in the first image to be annotated and the second image to be annotated according to the positions of each preset area identification point on the road surface.

[0062] The preset region of interest identification parameters are obtained based on the annotation results of the first annotated image and the second annotated image; wherein, the process of obtaining the preset region of interest identification parameters based on the annotation results can be understood as calculating the positional relationship of the region of interest relative to the scanned image (for example, the region of interest is a certain pixel interval range in the image) based on the position information of the pixel points corresponding to each identification point on the first scanned image and the second scanned image in the corresponding entire scanned image, as the corresponding preset region of interest identification parameters.

[0063] In practical applications, after the bird's-eye view conversion, The images at two focal lengths are used to crop the road area of ​​interest and remove invalid areas in the bird's-eye view, which can speed up the calculation and simplify the difficulty of the post-processing algorithm. Less than ,so The field of view is greater than 's vision, There are There are no road areas in Figure 5 As shown in the white area in (a), the road ROI area shared by the two cameras is as follows Figure 5 As shown in the dark area in (a); at the same time, since the first road scan image and the second road scan image share the same optical axis, according to Figure 6 From the dual-focus imaging principle shown in the figure, it can be seen that the first region of interest image has a first road surface matching constraint, and the first road surface matching constraint is a ray passing through the center point of the first region of interest image; the second region of interest image has a second road surface matching constraint, and the second road surface matching constraint is a ray passing through the center point of the second region of interest image; that is, the first region of interest image and the second region of interest image satisfy Figure 5 The road surface matching constraint shown in Figure (b) stipulates that the corresponding feature points of the first ROI image and the second ROI image must be on the ray passing through the center point of the regional image. The feature points can be subsequently aligned through the road surface matching constraint of the same optical axis to calculate the road surface scanning information.

[0064] S13, through polar coordinate conversion, obtaining a first polar coordinate region image based on the first region of interest image, and obtaining a second polar coordinate region image based on the second region of interest image; wherein the first polar coordinate region image and the second polar coordinate region image can be understood as considering Figure 5 The road surface matching constraint (ray constraint) shown in Figure (b) is too computationally intensive to directly use for registration. In order to simplify the registration algorithm and improve the registration efficiency, in some embodiments, the first region of interest image and the second region of interest image belonging to the Cartesian coordinate system are further converted into a polar coordinate system to obtain an image, which is then used for subsequent analysis and processing. The specific conversion process is as follows:

[0065] The first region of interest image and the second region of interest image are respectively expanded radially with the center of the corresponding region image as the center, the horizontal direction as the radius (the maximum value is R), and the vertical direction as the angle (the maximum is 360°). Specifically, the polar coordinate transformation formula is shown in the following formula, and the corresponding polar coordinate transformation method is as follows: Figure 7 As shown:

[0066]

[0067] in, is the center point of the image of the region of interest; is the point to be changed; is the distance between the center point and the point to be transformed; It is the angle between the road surface matching constraint ray and the horizontal line.

[0068] The effect of polar coordinate transformation of the road surface is as follows Figure 8 As shown, Figure 8 Figure (a) shows The whole area below, Figure 8 Figure (b) shows The lower road ROI area is zoomed to Image of the road surface ROI area size, Figure 8 Figure (c) shows Lower road surface ROI area, Figure 8 Figure (d) shows Figure 8 The polar coordinate region image of the middle (b) figure, Figure 8 Figure (e) shows Figure 8 The polar coordinate region image in the middle (c) figure; it can be seen that in the polar coordinate system, the road surface matching constraint ray is simultaneously transformed into a horizontal straight line constraint (similar to the polar line constraint), and the existing stereo matching strategy based on limit constraints can be referred to for alignment.

[0069] S14. Obtain the registration results of the first polar coordinate area image and the second polar coordinate area image, and obtain a feature point matching set based on the registration results; wherein, the process of obtaining the feature point matching set can be understood as a process of using an existing stereoscopic vision image matching strategy to perform registration to obtain all matching feature point pairs on the first region of interest image and the second region of interest image.

[0070] In some embodiments, the step of obtaining the registration result of the first polar coordinate area image and the second polar coordinate area image, and obtaining the feature point matching set according to the registration result includes:

[0071] According to the preset stereo registration method, based on the first polar coordinate area image and the second polar coordinate area image, the registration result is obtained; the registration result is a polar coordinate feature point matching set corresponding to the first polar coordinate area image and the second polar coordinate area image; wherein the preset stereo registration method is one of a feature similarity matching method based on horizontal straight line constraints and a stereo registration method based on deep learning, that is, the traditional disparity map matching algorithm and deep learning strategy can be used for registration. The corresponding feature similarity matching method based on horizontal straight line constraints can be understood as a similar disparity value matching strategy based on stereo vision with horizontal straight line constraints and feature point matching as matching principles, as follows Fig. 9 As shown, a set of points with the closest and most identical features are found at the same height position on the two polar coordinate area images, which are the matching feature point pairs. Fig. 9 The red box in the middle is the corresponding point box. The corresponding points of the left and right images are searched in the direction of the black dotted line (horizontal straight line constraint). The specific registration process is as follows:

[0072] According to the preset pixel sliding window, the feature similarities of all pixel points on the horizontal straight line constraints in the first polar coordinate area image and the second polar coordinate area image are calculated respectively; according to the feature similarity and maximum similarity registration principle, the polar coordinate feature point matching set is obtained.

[0073] In practical applications, it is assumed that the preset pixel sliding window is The pixel neighborhood window can use the SAD (Sum of Absolute Differences) algorithm to calculate the features of all points on the polar lines in the left and right images (the first polar coordinate area image and the second polar coordinate area image). and , is the pixel index of the left and right images, represents a set of positive integers, and the corresponding calculation formula is: ,in, , Pixel The preset pixel sliding window is calculated and Similarity Afterwards, according to Search for the registration point pair with the greatest similarity; finally, perform post-processing such as filtering on the obtained registration point pair, mainly using geometric imaging constraints. The matching point is closer to the center of the polar line in the short focal length image and closer to the image edge in the long focal length image to determine whether the match is correct, thereby obtaining the registration points of all pixels in the left image, that is, obtaining the required polar coordinate feature point matching set.

[0074] At the same time, the stereo registration method based on deep learning can be understood as performing feature extraction on the first polar coordinate area image and the second polar coordinate area image respectively through a preset convolutional neural network to obtain the corresponding first feature map and second feature map; according to the first feature map and the second feature map, based on a plurality of cost body construction methods, a plurality of cost bodies are obtained; according to a preset three-dimensional convolution structure, all cost bodies are cost aggregated, and parallax iterative learning is performed according to a preset configuration loss function to obtain the polar coordinate feature point matching set; it should be noted that the preset convolutional neural network and the preset three-dimensional convolution structure can be set according to actual application requirements, and no specific limitation is made here.

[0075] In practical applications, the deep learning-based registration process includes: a) using convolutional neural network to extract the first feature map And the second feature map ; b) According to the feature map, through feature correlation Connect with features The cost volume is constructed by the method of ; c) the cost aggregation is performed using the 3D convolution structure; d) the registration point is estimated using the L1 loss function shown in the following formula:

[0076]

[0077] in, is the total estimated number of pixels; The true registration point for each pixel; The predicted registration point for each pixel.

[0078] The final result of the above registration method is as follows Figure 8 Figure (d) and Figure 8 As shown in FIG. 5( e ), through registration, corresponding feature points P of the first polar coordinate region image and the second polar coordinate region image, ie, each matching polar coordinate feature point pair in the polar coordinate feature point matching set, can be obtained.

[0079] Through the inverse polar coordinate transformation, the feature point matching set is obtained based on the polar coordinate feature point matching set; wherein the feature point matching set can be understood as converting the corresponding polar coordinate feature point pairs in the polar coordinate system into matching feature point pairs in the Cartesian coordinate system of the original image, so as to facilitate mapping each matching feature point pair to the three-dimensional world coordinate system to obtain the three-dimensional information of the reconstructed road surface. Specifically, the inverse polar coordinate transformation is expressed as:

[0080]

[0081] Through the above inverse polar coordinate transformation, Figure 8 Figure (d) and Figure 8 The feature point P shown in Figure (e) is transformed into Figure 8Figure (b) and Figure 8 The feature point P shown in Figure (c) can be used to obtain each matching feature point pair in the Cartesian coordinate system.

[0082] In the above embodiment, the feature point registration is performed based on the road surface matching constraint of the same optical axis, which can ensure that the local features of the feature points are more unified, thereby effectively improving the matching efficiency and matching accuracy, and ensuring a better matching effect.

[0083] S15, obtaining a road area spatial position set according to the pixel position coordinates of each matching feature point pair in the feature point matching set; wherein the principle of dual-focus imaging is as follows Figure 6 As shown in the theoretical model based on pinhole imaging, a point in space , at two different focal lengths There are two projections on the imaging plane. According to the similar triangle theorem, the simultaneous formula is as follows:

[0084]

[0085] In the formula, For two focal lengths, are two projections of the image plane at the corresponding focal length, is the position of the plane point.

[0086] By extending the above planar imaging formula to the three-dimensional world coordinate system, we can obtain the calculation formula for the three-dimensional space:

[0087]

[0088] in, For space point Projection on the image plane, the coordinate point in space can be calculated according to the formula .

[0089] In some embodiments, the step of obtaining a road surface area spatial position set according to the pixel position coordinates of each matching feature point pair in the feature point matching set includes:

[0090] Through the principle of dual-focus imaging, based on the pixel position coordinates of each matching feature point pair in the feature point matching set, the spatial position coordinates of the road surface area are obtained; wherein the spatial position coordinates of the road surface area can be understood as the focal length of the lens corresponding to the first road surface scanning image and the second road surface scanning image, and the pixel position coordinates of each matching feature point pair in the feature point matching set are converted into three-dimensional space to obtain the spatial position of each point on the road surface in the world coordinate system. Correspondingly, the preset spatial coordinate conversion formula is expressed as:

[0091]

[0092] in, and Respectively represent the pixel position coordinates of the i-th matching feature point pair in the feature point matching set corresponding to the first region of interest image and the second region of interest image; Represents the spatial position coordinates of the road surface area corresponding to the i-th matching feature point pair; and They respectively represent the focal lengths of the lenses used to acquire the first road surface scanning image and the second road surface scanning image.

[0093] The above method can be used to calculate the spatial positions of all pixel points in the entire road surface ROI area, that is, to obtain an accurate set of road surface area spatial positions, providing reliable data support for subsequent road surface 3D reconstruction.

[0094] S16. Obtain a road surface preview area information set based on the road surface area spatial position set through vehicle body coordinate system conversion; wherein the road surface preview area information set can be understood as a coordinate system conversion matrix constructed based on the positional relationship between the binocular camera for obtaining the first road surface scanning image and the second road surface scanning image and the vehicle body coordinate system, and the road surface position information set is obtained by converting each spatial position in the road surface area spatial position set into position information under the vehicle body coordinate system.

[0095] S17. Obtain a three-dimensional reconstruction result of the road surface according to the road surface preview area information set; wherein, the three-dimensional reconstruction result of the road surface can be understood as a result obtained by performing three-dimensional reconstruction of the road surface according to the road surface preview area information set. The specific three-dimensional reconstruction method of the road surface can be implemented by referring to the existing technology and will not be described in detail here.

[0096] In this embodiment, a first road surface scanning image and a second road surface scanning image with a common optical axis are obtained by using a binocular camera, and regions of interest are extracted from the first road surface scanning image and the second road surface scanning image to obtain first and second regions of interest images corresponding to road surface matching constraints as rays passing through the center points of the regions of interest images. Polar coordinate conversion is performed on the first and second regions of interest images with the centers of the corresponding regions of interest as the centers of the circles to obtain corresponding first polar coordinate region images and second polar coordinate region images, the road surface matching constraints are converted into corresponding horizontal straight line constraints, and feature point matching is performed on the first and second polar coordinate region images to obtain The corresponding feature point matching set, according to the pixel position coordinates of each matching feature point pair in the feature point matching set, obtains the corresponding road area spatial position set based on the dual-focus imaging principle, and converts each spatial position in the road area spatial position set into the vehicle body coordinate system to obtain the corresponding road preview area information set, and performs a technical solution for road surface three-dimensional reconstruction based on the road surface preview area information set. By performing binocular road stereo image alignment based on the road surface matching constraint based on the same optical axis, it can not only avoid excessive calculation operations and ensure the simplicity and efficiency of the alignment process, but also effectively improve the alignment accuracy and alignment effect, providing a reliable basis for road surface three-dimensional reconstruction, and then providing a reliable guidance basis for the formulation of intelligent driving strategies.

[0097] It should be noted that although the steps in the above flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders.

[0098] In some embodiments, Fig.10 As shown, a binocular camera is provided, the binocular camera includes a first camera body 11, a second camera body 12, a first lens 13, a second lens 14, a semi-transparent and semi-reflective mirror 15 and a reflector 16;

[0099] The semi-transparent and semi-reflective mirror 15 has a light-transmitting end aligned with the light-incident end of the reflector 16, and a light-reflecting end aligned with the light-incident end of the second lens 14, and the semi-transparent and semi-reflective mirror 15 is used to reflect and transmit the received incident light to obtain corresponding first reflected light and transmitted light;

[0100] The second lens 14 has a light-emitting end aligned with a light-incoming end of the second camera body 12, and the second lens 14 is used to converge the received first reflected light and emit it into the second camera body 12;

[0101] The reflector 16 has a light output end aligned with the light input end of the first lens 13, and is used to reflect the received transmitted light to obtain a second reflected light parallel to the first reflected light;

[0102] The light emitting end of the first lens 13 is aligned with the light incident end of the first camera body 11 , and the first lens 13 is used to converge the received second reflected light and emit it into the first camera body 11 .

[0103] In some embodiments, Fig.10 As shown, the first camera body 11 and the second camera body 12 are arranged on the same side, and the mirror surfaces of the semi-transparent and semi-reflective mirror 15 and the reflector 16 are kept parallel. After the incident light enters the semi-transparent and semi-reflective mirror 15 (beam splitter) with a projection and reflection ratio of 50:50, a beam of light is reflected to form a first reflected light and enters the second lens 14 (lens or lens group) with a focal length of f2, and is converged by the second lens 14 to enter the second camera body 12 for imaging to generate a first road surface scanning image; at the same time, another beam of light enters the reflector 16 through transmission, and is reflected by the reflector 16 to generate a second reflected light parallel to the first reflected light and enters the first lens 13 with a focal length of f1, and is converged by the first lens 13 (lens or lens group) to enter the first camera body 11 for imaging, thereby generating a second road surface scanning image having the same optical axis as the first road surface scanning image. It should be noted that the specific types of the first camera body 11, the second camera body 12, the first lens 13, the second lens 14 and the reflector 16 can be selected and set according to actual application requirements, among which the reflector 16 preferably adopts a 45-degree reflector with high reflectivity, wide reflection frequency band and good density to provide reliable guarantee for the imaging quality of the second camera body 12.

[0104] In some embodiments, the binocular camera can also be used Fig.11 In the arrangement structure shown, the first camera body 11 and the second camera body 12 are arranged on both sides respectively, and the mirror surfaces of the semi-transparent and semi-reflective mirror 15 and the reflector 16 are kept in a vertical state; it can also be achieved that after the incident light enters the semi-transparent and semi-reflective mirror 15 (beam splitter) with a projection and reflection ratio of 50:50, a beam of light is reflected to form a first reflected light and enters the second lens 14 (lens or lens group) with a focal length of f2, and is converged by the second lens 14 to enter the second camera body 12 for imaging to generate a first road surface scanning image; at the same time, another beam of light enters the reflector 16 through transmission, and is reflected by the reflector 16 to generate a second reflected light parallel to the first reflected light and enters the first lens 13 with a focal length of f1, and is converged by the first lens 13 (lens or lens group) to enter the first camera body 11 for imaging, thereby generating an effect of a second road surface scanning image having the same optical axis as the first road surface scanning image.

[0105] based on Fig.10 and Fig.11 It can be seen from the layout structure of the binocular camera shown that the first camera body 11, the second camera body 12, the first lens 13, the second lens 14, the semi-transparent and semi-reflective mirror 15 and the reflector 16 in the binocular camera only need to meet the corresponding optical path design requirements to generate the first road surface scanning image and the second road surface scanning image of the same optical axis. In some embodiments, the semi-transparent and semi-reflective mirror 15 is used to receive the incident light, and reflect and transmit the incident light to obtain the corresponding first reflected light and transmitted light; the second lens 14 is used to receive the first reflected light so that the first reflected light merges into the second camera body 12; the reflector 16 is used to receive the transmitted light and reflect to generate the corresponding second reflected light so that the second reflected light is parallel to the first reflected light; the first lens 13 is used to receive the second reflected light so that the second reflected light merges into the first camera body 11.

[0106] In order to minimize the imaging differences caused by factors other than the location of the binocular camera in actual applications, in some embodiments, the target surface size and pixel size of the first camera body 11 and the second camera body 12 are further set to be the same.

[0107] At the same time, considering that zooming and adjusting the camera's field of view may cause shooting delays, in order to maximize imaging efficiency, in some embodiments, the focal lengths of the first lens 13 and the second lens 14 are further set to be different. In addition, in some embodiments, the focal lengths of the first lens 13 and the second lens 14 can be adjusted to adapt to the road preview requirements of different distance ranges, and can effectively ensure the corresponding road preview accuracy, obtain high-quality road scan images, and provide a reliable analysis basis for subsequent road 3D reconstruction processing.

[0108] The binocular camera provided in the above embodiment realizes the common path of the optical axes of binocular cameras of different focal lengths through the combined design of semi-transparent and semi-reflective and reflective mirrors, which not only increases the camera field of view, but also effectively avoids the camera field of view being affected by the baseline distance, and can also provide binocular road scanning images with a common optical axis, providing reliable support for simple, efficient and accurate binocular image stereo registration based on the road matching constraint of the same optical axis, thereby effectively improving the efficiency and accuracy of three-dimensional reconstruction of the road surface.

[0109] In practical applications, the binocular camera provided in the above embodiment can be directly arranged on the vehicle body as a sensor for the vehicle vision system to collect road scanning images, and the road scanning images in front of the vehicle road collected in real time can be used for corresponding road image analysis and detection; there are two ways to arrange the binocular camera: Fig.12The binocular dual-focus camera shown in Figure (a) is arranged horizontally so that the optical axis of the camera sensor (the optical axis of the first lens and the second lens) is parallel to the road surface. Fig.12 The binocular dual-focus camera shown in Figure (b) is arranged downwardly in such a way that the optical axis of the camera sensor (the optical axis of the first lens and the second lens) intersects with the road surface. Considering that the arrangement of the camera sensor optical axis parallel to the road surface is more conducive to binocular perception, but there will be fewer pixels of the road surface in the image, and the arrangement of the camera sensor optical axis intersecting with the road surface is more conducive to road scanning, with more road surface pixels in the image and higher resolution, in some embodiments, the optical axis of the first lens and the second lens is further arranged to intersect with the road surface, so as to facilitate the acquisition of images with more road surface pixels, which is conducive to the subsequent road surface pre-aiming analysis and processing; at the same time, when the binocular dual-focus camera is arranged downwardly in an inclined manner, the specific angle at which the optical axis of the camera sensor intersects with the road surface can be used to balance the pre-scanning road surface range (especially the length of the road surface ahead), which is related to the actual required support of the binocular perception function of autonomous driving, and there is no specific angle parameter restriction here.

[0110] The binocular camera provided in the above embodiment can be understood as a binocular bifocal ranging principle based on pinhole imaging. The binocular bifocal stereo vision perception system with a common optical axis realized by introducing a design combination of a semi-transparent and semi-reflective mirror and a reflector can be installed on a vehicle or other movable equipment for pre-aiming detection of the driving road surface. It can increase the camera field of view and effectively avoid the field of view being affected by the baseline distance. At the same time, it can provide a binocular road scanning image with a common optical axis, which is convenient for realizing a simple, efficient and accurate binocular image stereo registration based on the road surface matching constraint of the same optical axis.

[0111] In one embodiment, Fig.13 As shown, a road preview system is provided, the system comprising an image acquisition module 1, a region extraction module 2, a polar coordinate conversion module 3, a stereo registration module 4, a spatial information acquisition module 5, a preview information acquisition module 6 and a three-dimensional reconstruction module 7;

[0112] The image acquisition module 1 is used to obtain a first road surface scanning image and a second road surface scanning image through a binocular camera; the first road surface scanning image and the second road surface scanning image are co-optical axis images;

[0113] The region extraction module 2 is used to obtain a first region of interest image based on the first road surface scanning image, and to obtain a second region of interest image based on the second road surface scanning image;

[0114] The polar coordinate conversion module 3 is used to obtain a first polar coordinate region image based on the first region of interest image and obtain a second polar coordinate region image based on the second region of interest image through polar coordinate conversion;

[0115] The stereo registration module 4 is used to obtain the registration result of the first polar coordinate area image and the second polar coordinate area image, and obtain a feature point matching set according to the registration result;

[0116] The spatial information acquisition module 5 is used to obtain a road surface area spatial position set according to the pixel position coordinates of each matching feature point pair in the feature point matching set;

[0117] The preview information acquisition module 6 is used to obtain a road preview area information set based on the road area spatial position set by converting the vehicle body coordinate system;

[0118] The three-dimensional reconstruction module 7 is used to obtain a three-dimensional reconstruction result of the road surface according to the road surface preview area information set.

[0119] For the specific definition of the road preview system, please refer to the definition of the road preview method above, and the corresponding technical effects can also be obtained equivalently, which will not be repeated here. Each module in the above-mentioned road preview system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0120] In some embodiments, a car is provided on which a computer device is deployed. The computer device includes: a memory, a processor, and a road surface preview program stored in the memory and executable on the processor, wherein the road surface preview program is configured with steps for implementing the above-mentioned road surface preview method.

[0121] In some embodiments, a controller is provided, characterized in that the controller includes a processor and a memory, the processor and the memory are coupled, the memory is used for computer programs or instructions, and the processor is used to execute the computer programs or instructions in the memory, so that the controller performs the steps of the above-mentioned road surface preview method.

[0122] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned road surface preview method are implemented.

[0123] In summary, the embodiments of the present invention provide a preview method, a binocular camera, a preview system, a car, a controller and a medium. Compared with the prior art, the road preview method performs binocular road stereo image registration through road matching constraints based on the same optical axis, which can not only avoid excessive calculation operations and ensure the simplicity and efficiency of the registration process, but also effectively improve the registration accuracy and registration effect, provide a reliable basis for three-dimensional reconstruction of the road surface, and further provide a reliable guidance basis for the formulation of intelligent driving strategies; the binocular camera realizes the common path of the optical axes of binocular cameras of different focal lengths through the combination design of semi-transparent and semi-reflective and reflective mirrors, which can not only increase the camera field of view, but also effectively avoid the camera field of view being affected by the baseline distance, and can also provide binocular road scanning images with the same optical axis, so as to provide reliable support for simple, efficient and accurate binocular image stereo registration based on the road matching constraints of the same optical axis, thereby effectively improving the efficiency and accuracy of three-dimensional reconstruction of the road surface.

[0124] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. A road preview method, characterized in that: The method comprises the following steps: Acquire a first road surface scanning image and a second road surface scanning image through a binocular camera; the first road surface scanning image and the second road surface scanning image are co-optical axis images; Obtain a first region of interest image based on the first road surface scan image, and obtain a second region of interest image based on the second road surface scan image; By polar coordinate conversion, a first polar coordinate region image is obtained based on the first region of interest image, and a second polar coordinate region image is obtained based on the second region of interest image; Acquire a registration result of the first polar coordinate region image and the second polar coordinate region image, and obtain a feature point matching set according to the registration result; Obtaining a road area spatial position set according to the pixel position coordinates corresponding to each matching feature point in the feature point matching set; Obtaining a road surface preview area information set based on the road surface area spatial position set by converting the vehicle body coordinate system; A three-dimensional reconstruction result of the road surface is obtained according to the road surface preview area information set.

2. The road surface preview method according to claim 1, characterized in that: The binocular camera includes a first camera body, a second camera body, a first lens, a second lens, a semi-transparent and semi-reflective mirror, and a reflector; The semi-transparent and semi-reflective mirror has a light-transmitting end aligned with the light-incident end of the reflector and a light-reflecting end aligned with the light-incident end of the second lens, and the semi-transparent and semi-reflective mirror is used to reflect and transmit the received incident light to obtain corresponding first reflected light and transmitted light; The second lens has a light output end aligned with a light input end of the second camera body, and the second lens is used to converge the received first reflected light and emit it into the second camera body; The reflector has a light output end aligned with the light input end of the first lens, and is used to reflect the received transmitted light to obtain a second reflected light parallel to the first reflected light; The light emitting end of the first lens is aligned with the light incident end of the first camera body, and the first lens is used to converge the received second reflected light and emit it into the first camera body.

3. The road surface preview method according to claim 1, characterized in that: The first region of interest image has a first road surface matching constraint, which is a ray passing through the center point of the first region of interest image; the second region of interest image has a second road surface matching constraint, which is a ray passing through the center point of the second region of interest image.

4. The road surface preview method according to claim 1, characterized in that: The step of obtaining a first region of interest image based on the first road surface scan image and obtaining a second region of interest image based on the second road surface scan image comprises: By using the affine transformation principle, a first road surface bird's-eye view is obtained based on the first road surface scan image, and a second road surface bird's-eye view is obtained based on the second road surface scan image; According to preset region of interest identification parameters, a first region of interest image is obtained based on the first road surface bird's-eye view, and a second region of interest image is obtained based on the second road surface bird's-eye view.

5. The road surface preview method according to claim 4, characterized in that: The step of obtaining the preset region of interest identification parameters includes: Acquire a first image to be annotated and a second image to be annotated including a preset area identification point; According to the preset area identification points, a first annotated image is obtained based on the first image to be annotated, and a second annotated image is obtained based on the second image to be annotated; The preset region of interest recognition parameters are obtained according to the annotation results of the first annotated image and the second annotated image.

6. The road surface preview method according to claim 1, characterized in that: The step of obtaining a first polar coordinate region image based on the first region of interest image and obtaining a second polar coordinate region image based on the second region of interest image through polar coordinate conversion comprises: Taking the region image center of the first region of interest image as the center of a circle, obtaining a first polar coordinate region image corresponding to the first region of interest image based on polar coordinate transformation; Taking the region image center of the second region of interest image as the center of a circle, a second polar coordinate region image corresponding to the second region of interest image is obtained based on polar coordinate transformation.

7. The road surface preview method according to claim 6, characterized in that: The road surface matching constraint between the first polar coordinate area image and the second polar coordinate area image is a horizontal straight line constraint.

8. The road surface preview method according to claim 1, characterized in that: The step of obtaining the registration result of the first polar coordinate area image and the second polar coordinate area image, and obtaining a feature point matching set according to the registration result comprises: According to a preset stereo registration method, based on the first polar coordinate region image and the second polar coordinate region image, the registration result is obtained; the registration result is a matching set of polar coordinate feature points corresponding to the first polar coordinate region image and the second polar coordinate region image; The feature point matching set is obtained based on the polar coordinate feature point matching set through inverse polar coordinate transformation.

9. The road surface preview method according to claim 8, characterized in that: The preset stereo registration method is one of a feature similarity matching method based on road surface matching constraints and a stereo registration method based on deep learning.

10. The road surface preview method according to claim 1, characterized in that: The step of obtaining a road surface area spatial position set according to the pixel position coordinates of each matching feature point pair in the feature point matching set comprises: Through the principle of dual-focus imaging, the spatial position coordinates of the road surface area are obtained based on the pixel position coordinates of each matching feature point pair in the feature point matching set.

11. A binocular camera, characterized in that: The binocular camera includes a first camera body, a second camera body, a first lens, a second lens, a semi-transparent and semi-reflective mirror, and a reflector; The semi-transparent and semi-reflective mirror has a light-transmitting end aligned with the light-incident end of the reflector and a light-reflecting end aligned with the light-incident end of the second lens, and the semi-transparent and semi-reflective mirror is used to reflect and transmit the received incident light to obtain corresponding first reflected light and transmitted light; The second lens has a light output end aligned with a light input end of the second camera body, and the second lens is used to converge the received first reflected light and emit it into the second camera body; The reflector has a light output end aligned with the light input end of the first lens, and is used to reflect the received transmitted light to obtain a second reflected light parallel to the first reflected light; The light emitting end of the first lens is aligned with the light incident end of the first camera body, and the first lens is used to converge the received second reflected light and emit it into the first camera body.

12. The binocular camera according to claim 11, wherein: The first camera body and the second camera body have the same target surface size and pixel size.

13. The binocular camera according to claim 11, characterized in that: The first lens and the second lens have different focal lengths.

14. A road preview system, characterized in that: The system includes an image acquisition module, a region extraction module, a polar coordinate conversion module, a stereo registration module, a spatial information acquisition module, a preview information acquisition module and a three-dimensional reconstruction module; The image acquisition module is used to obtain a first road surface scanning image and a second road surface scanning image through a binocular camera; the first road surface scanning image and the second road surface scanning image are co-optical axis images; The region extraction module is used to obtain a first region of interest image based on the first road surface scanning image, and to obtain a second region of interest image based on the second road surface scanning image; The polar coordinate conversion module is used to obtain a first polar coordinate region image based on the first region of interest image and obtain a second polar coordinate region image based on the second region of interest image through polar coordinate conversion; The stereo registration module is used to obtain the registration result of the first polar coordinate area image and the second polar coordinate area image, and obtain a feature point matching set according to the registration result; The spatial information acquisition module is used to obtain a road surface area spatial position set according to the pixel position coordinates of each matching feature point pair in the feature point matching set; The preview information acquisition module is used to obtain a road surface preview area information set based on the road surface area spatial position set through vehicle body coordinate system conversion; The three-dimensional reconstruction module is used to obtain a three-dimensional reconstruction result of the road surface according to the road surface preview area information set.

15. A car, characterized in that: A computer device is deployed thereon, the computer device comprising: a memory, a processor, and a road surface preview program stored in the memory and executable on the processor, the road surface preview program being configured with steps for implementing the road surface preview method as described in any one of claims 1 to 10.

16. A controller, characterized in that: The controller includes a processor and a memory, the processor and the memory are coupled, the memory is used for computer programs or instructions, and the processor is used to execute the computer programs or instructions in the memory, so that the controller performs the steps of the road surface preview method as described in any one of claims 1 to 10.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the road surface preview method described in any one of claims 1 to 10 are implemented.

Images