Interframe matching detection method, system and intelligent terminal based on binocular vision system

Abstract

The application discloses a kind of interframe matching detection method, system and intelligent terminal based on binocular vision system, the method includes: respectively obtaining the overhead gray diagram and overhead segmentation diagram of two adjacent frames before and after in same road scene;Through the overhead segmentation diagram, the matching candidate region of previous frame is detected;Through vehicle speed and time stamp, the initial moving distance between two frames and the initial estimated position of the next frame are calculated, and based on the initial estimated position of the next frame, the search area of the next frame is obtained;The matching candidate region is matched with the search area, and the matching position deviation is calculated;The initial moving distance is corrected using the matching position deviation, to obtain interframe matching result.It improves the registration accuracy of data between adjacent frames in image processing in auxiliary driving, and further provides more accurate image processing data for auxiliary driving system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving assistance, in particular to a frame matching detection method and system based on a binocular vision system and an intelligent terminal. BACKGROUND

[0002] With the development of automatic driving technology, people's requirements for the safety and comfort of assisted driving vehicles are also increasing. In assisted driving, the effect of image processing, especially the data registration accuracy between adjacent frames, has a greater impact on the control effect of assisted driving, and is directly related to the safety and comfort of assisted driving vehicles.

[0003] Therefore, to improve the data registration accuracy between adjacent frames in image processing of assisted driving and provide more accurate image processing data for the assisted driving system, a frame matching detection method based on a binocular vision system is provided, which has become a problem to be solved by those skilled in the art. SUMMARY

[0004] To this end, the embodiments of the present application provide a frame matching detection method and system based on a binocular vision system and an intelligent terminal, so as to improve the data registration accuracy between adjacent frames in image processing of assisted driving and provide more accurate image processing data for the assisted driving system.

[0005] In order to achieve the above-mentioned purpose, the embodiments of the present application provide the following technical solutions:

[0006] The present application provides a frame matching detection method based on a binocular vision system, characterized in that the method comprises:

[0007] Respectively acquiring an overhead gray image and an overhead segmentation image of two adjacent frames before and after a same road scene;

[0008] Detecting a candidate matching region of a previous frame through the overhead segmentation image;

[0009] Calculating an initial moving distance between the two frames and an initial estimated position of a next frame through a vehicle speed and a time stamp, and acquiring a search region of the next frame based on the initial estimated position of the next frame;

[0010] Making template matching between the candidate matching region and the search region, and calculating a matching position deviation;

[0011] Correcting the initial moving distance by using the matching position deviation to obtain a frame matching result.

[0012] Further, the method of respectively acquiring an overhead gray image and an overhead segmentation image of two adjacent frames before and after a same road scene specifically comprises:

[0013] Obtaining left and right views of the same road scene, and processing the left and right views to obtain a dense disparity map of the road scene;

[0014] Based on the dense disparity map, converting image information of a target region into three-dimensional point cloud information in a world coordinate system, and fitting a road surface model based on the three-dimensional point cloud information;

[0015] In the dense disparity map, a target region is demarcated, the image of the target region is input into a trained semantic segmentation model, and two-dimensional image information after segmentation is obtained;

[0016] Based on the two-dimensional image information and the three-dimensional point cloud information, the gray image of the detection region is converted to the XOZ projection plane to generate the overhead gray image, and the segmentation image of the detection region is converted to the XOZ projection plane to generate the overhead segmentation image.

[0017] Further, the conversion of the image information of the target region into the three-dimensional point cloud information in the world coordinate system based on the dense disparity map specifically includes:

[0018] Based on a binocular stereo vision system imaging model and a pinhole imaging model, the image coordinate system of the dense disparity map is converted into a world coordinate system;

[0019] Taking the target region in the real world coordinate system as a reference, the target region is intercepted in the dense disparity map;

[0020] The image information in the target region is converted into three-dimensional point cloud information according to the following formula:

[0021] Z = B * F / disp

[0022] X = (Img x -cx) * b / disp

[0023] Y = (Img y -cy) * b / disp

[0024] Wherein, B is the distance between the optical center of the left camera and the optical center of the right camera in the binocular stereo vision imaging system;

[0025] F is the focal length of the camera in the binocular stereo vision imaging system;

[0026] cx and cy are the image coordinates of the principal point of the camera in the binocular stereo vision imaging system;

[0027] Img x and Img y are image coordinate points in the target region;

[0028] disp is the image point coordinatex , Img y ) of the disparity value;

[0029] X is the horizontal distance of the three-dimensional point from the camera in the world coordinate system;

[0030] Y is the vertical distance of the three-dimensional point from the camera in the world coordinate system;

[0031] Z is the depth distance of the three-dimensional point from the camera in the world coordinate system.

[0032] Further, the road surface model equation fitted based on the three-dimensional point cloud information is:

[0033] cosα*X+cosβ*Y+cosγ*Z+D=0

[0034] Wherein, cosα is the direction cosine of the angle between the road surface normal vector and the x coordinate axis of the world coordinate system;

[0035] cosβ is the direction cosine of the angle between the road surface normal vector and the y coordinate axis of the world coordinate system;

[0036] cosγ is the direction cosine of the angle between the road surface normal vector and the z coordinate axis of the world coordinate system;

[0037] D is the distance from the origin of the world coordinate system to the road surface.

[0038] Further, the following formula is used to complete the homography transformation based on the two-dimensional image information and the three-dimensional point cloud information:

[0039]

[0040] Proj x =X*scale x

[0041] Proj x =Z*scale z

[0042] Wherein, X and Z are the horizontal distance and the depth distance from the camera respectively;

[0043] Img x and Img y are the image coordinate points in the detection area;

[0044] Proj x and Proj y are the projection surface coordinate points in the detection area;

[0045] H is the homography transformation matrix.

[0046] Further, the detecting the candidate region to be matched of the previous frame through the top-view segmentation map specifically comprises:

[0047] The pixel value on the top-view segmentation map of the previous frame is marked as a preset mark pixel value, and other positions are set as 0, and a binary image is generated therefrom;

[0048] The connected domain is detected on the binary image, and a region of a preset size is expanded around the lower boundary center position of the connected domain as a candidate region.

[0049] Further, the obtaining the search region of the next frame based on the initial estimated position of the next frame specifically comprises:

[0050] The initial estimated position of the candidate region in the next frame is obtained based on the initial moving distance and the candidate region;

[0051] The search region is expanded around the initial estimated position by a preset size.

[0052] The application further provides a frame inter-matching detection system based on a binocular vision system, which comprises:

[0053] An image acquisition unit is configured to acquire top-view gray-scale images and top-view segmentation maps of two adjacent frames before and after a same road scene respectively;

[0054] A candidate region acquisition unit is configured to detect a candidate region to be matched of a previous frame through the top-view segmentation map;

[0055] A search region acquisition unit is configured to calculate an initial moving distance between the two frames and an initial estimated position of a next frame through a vehicle speed and a time stamp, and obtain a search region of the next frame based on the initial estimated position of the next frame;

[0056] A position deviation acquisition unit is configured to perform template matching on the candidate region to be matched and the search region, and calculate a matching position deviation;

[0057] A matching result output unit is configured to correct the initial moving distance by using the matching position deviation, so as to obtain a frame inter-matching result.

[0058] The application further provides an intelligent terminal, which comprises a data acquisition device, a processor and a memory.

[0059] The data acquisition device is configured to acquire data; the memory is configured to store one or more program instructions; and the processor is configured to execute the one or more program instructions to execute the method as described above.

[0060] The application also provides a computer readable storage medium, wherein the computer storage medium comprises one or more program instructions for executing the method described above.

[0061] The method for inter-frame matching detection based on a binocular vision system provided by the application obtains a top-view gray image and a top-view segmentation image of two adjacent frames before and after a same road scene, detects a candidate region to be matched of a previous frame through the top-view segmentation image, calculates an initial moving distance between the two frames and an initial estimated position of a next frame through a vehicle speed and a time stamp, and obtains a search region of the next frame based on the initial estimated position of the next frame; performs template matching on the candidate region to be matched and the search region, calculates a matching position deviation, and finally corrects the initial moving distance by using the matching position deviation to obtain an inter-frame matching result. In this way, the method obtains a position deviation through matching of the candidate region to be matched and the search region, and corrects and compensates the initial moving distance by using the position deviation, so that the registration accuracy between the two adjacent frames is high, the registration accuracy of data between adjacent frames in image processing of the auxiliary driving is improved, and more accurate image processing data is provided for the auxiliary driving system. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can be obtained from the provided drawings without creative labor.

[0063] The structures, proportions, sizes, etc. shown in the specification are only used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and do not define the limiting conditions for the implementation of the application, so they do not have technical substantive significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effects and purposes that can be achieved by the application, should still fall within the scope of the technical content disclosed by the application.

[0064] Figure 1 A flowchart of a specific embodiment of the method for inter-frame matching detection based on a binocular vision system provided by the application;

[0065] Figure 2 A structural block diagram of a specific embodiment of the system for inter-frame matching detection based on a binocular vision system provided by the application. DETAILED DESCRIPTION

[0066] The following embodiments of the present application are explained by way of specific examples, and other advantages and effects of the present application will be readily appreciated by skilled in the art from the following description. It is obvious that the described embodiments are a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0067] The inter-frame matching detection method based on the binocular vision system can improve the registration accuracy of data between adjacent frames in image processing of assisted driving.

[0068] In one embodiment, as shown in Figure 1 The inter-frame matching detection method provided by the present application comprises the following steps:

[0069] S1: Obtain the top-view gray image and the top-view segmentation image of the adjacent two frames before and after the same road scene respectively.

[0070] In step S1, the top-view gray image and the top-view segmentation image of the adjacent two frames before and after the same road scene are obtained respectively, specifically including:

[0071] S11: Obtain the left and right views of the same road scene, and process the left and right views to obtain the dense disparity map of the road scene.

[0072] That is, the left and right views of the same road scene are obtained by a binocular stereo vision sensor, and the left and right views are processed to obtain the dense disparity map of the road scene.

[0073] In this embodiment, the binocular stereo camera coordinate system is taken as the reference system, the Z-axis distance direction is along the optical axis direction of the left camera, the X-axis horizontal direction is the binocular camera baseline direction, and the Y-axis vertical direction is the vertical direction.

[0074] S12: Based on the dense disparity map, the image information of the target region is converted into three-dimensional point cloud information in the world coordinate system, and a road surface model is fitted based on the three-dimensional point cloud information.

[0075] Specifically, the target region in the image is intercepted based on the target region in the real world coordinate system, and the image region of the target region is converted into three-dimensional point cloud information pts in the world coordinate system; the image region information is converted from the image coordinate system to the world coordinate system according to the binocular stereo vision system imaging model and the pinhole imaging model.

[0076] In order to improve the accuracy of three-dimensional point cloud information, and further ensure the accuracy of subsequent calculation results, in step S12, the image information of the target region is converted into three-dimensional point cloud information in the world coordinate system based on the dense disparity map, specifically including:

[0077] Based on the binocular stereo vision system imaging model and the pinhole imaging model, the image coordinate system of the dense disparity map is converted into the world coordinate system;

[0078] Taking the target region in the real world coordinate system as a reference, the target region is intercepted in the dense disparity map;

[0079] According to the following formula, the image information in the target region is converted into three-dimensional point cloud information:

[0080] Z=B*F / disp

[0081] X=(Img x -cx)*b / disp

[0082] Y=(Img y -cy)*b / disp

[0083] Wherein, B is the distance between the optical center of the left camera and the optical center of the right camera in the binocular stereo vision imaging system;

[0084] F is the focal length of the camera in the binocular stereo vision imaging system;

[0085] cx and cy are the image coordinates of the principal point of the camera in the binocular stereo vision imaging system;

[0086] Img x and Img y are the image coordinate points in the target region;

[0087] disp is the disparity value of the image point coordinate (Img x , Img y );

[0088] X is the horizontal distance of the three-dimensional point from the camera in the world coordinate system;

[0089] Y is the vertical distance of the three-dimensional point from the camera in the world coordinate system;

[0090] Z is the depth distance of the three-dimensional point from the camera in the world coordinate system.

[0091] In step S12, the road surface model equation fitted based on the three-dimensional point cloud information is:

[0092] cosα*X+cosβ*Y+cosγ*Z+D=0

[0093] wherein cos a is the direction cosine of the angle between the road surface normal vector and the x-axis of the world coordinate system;

[0094] cos β is the direction cosine of the angle between the road surface normal vector and the y-axis of the world coordinate system;

[0095] cos γ is the direction cosine of the angle between the road surface normal vector and the z-axis of the world coordinate system;

[0096] D is the distance from the origin of the world coordinate system to the road surface.

[0097] S13: A target area is demarcated in the dense disparity map, an image of the target area is input into a trained semantic segmentation model, and segmented two-dimensional image information is obtained.

[0098] To obtain an accurate semantic segmentation model, the terrain conditions that can occur in the road can be analyzed first, the common scene categories of the terrain are classified, and then each type of scene is photographed to obtain multiple training images. Then, for each training image, a mask image is obtained by labeling in the region of interest. For example, the pixel value of the bridge joint is labeled as 0, the pixel value of the ordinary road surface is labeled as 1, the pixel value of the road marking is labeled as 2, the pixel value of the speed bump is labeled as 3, the pixel value of the manhole cover is labeled as 4, and the pixel value of the accumulated water is labeled as 5. Thus, a mask image corresponding to each training image can be obtained.

[0099] S14: Based on the two-dimensional image information and the three-dimensional point cloud information, a homography transformation is performed to convert the gray image of the detection area to the XOZ projection plane to generate the overhead gray image, and to convert the segmentation image of the detection area to the XOZ projection plane to generate the overhead segmentation image.

[0100] In step S14, the homography transformation based on the two-dimensional image information and the three-dimensional point cloud information is completed using the following formula:

[0101]

[0102] Proj x = X * scale x

[0103] Proj x = Z * scale z

[0104] wherein X and Z are the horizontal distance from the camera and the depth from the camera, respectively;

[0105] Img x and Img y are the image coordinate points in the detection area;

[0106] Projx and Proj y is a projection plane coordinate point in the detection region;

[0107] H is the homographic transformation matrix.

[0108] The gray image of the detection region can be converted to the XOZ projection plane by homographic transformation to generate a top-down gray image, and the segmentation image of the detection region can be converted to the XOZ projection plane to generate a top-down segmentation image.

[0109] S2: detecting a candidate region to be matched of the previous frame through the top-down segmentation image.

[0110] It should be understood that in this specific embodiment, the previous frame refers to the frame in front of the two adjacent frames, and for the convenience of description, the previous frame is set as the t-1 frame; the next frame refers to the frame behind the two adjacent frames, and for the convenience of description, the next frame is set as the t frame.

[0111] In step S2, in order to improve the accuracy of the delineation of the candidate region, the candidate region to be matched of the previous frame is detected through the top-down segmentation image, specifically including the following steps:

[0112] S21: setting the pixel value marked as a preset marker on the top-down segmentation image of the previous frame to 1, and setting other positions to 0, and generating a binary image therefrom;

[0113] S22: detecting a connected domain on the binary image, and expanding a region of a preset size around the center position of the lower boundary of the connected domain as a candidate region.

[0114] For example, in one specific use scenario, the candidate region with pixel value marked as 2, 3 and 4 is detected on the t-1 frame top-down segmentation image, specifically, the pixel value marked as 2, 3 and 4 on the t-1 frame top-down segmentation image is set to 1, and other positions are set to 0 to generate a binary image, and a connected domain is detected on the binary image, and a region of a fixed size is expanded around the center position of the lower boundary of the connected domain as a candidate region.

[0115] S3: calculating an initial moving distance between the two frames and an initial estimated position of the next frame through the vehicle speed and the time stamp, and obtaining a search region of the next frame based on the initial estimated position of the next frame;

[0116] The search region of the next frame is obtained based on the initial estimated position of the next frame, specifically including the following steps:

[0117] S31: obtaining an initial estimated position of the candidate region in the next frame based on the initial moving distance and the candidate region;

[0118] S32: Expanding a region of a preset size around the initial estimated position as a search region.

[0119] In the specific use scenario described above, the initial movement distance between the two frames is calculated by the vehicle speed information, the t-frame timestamp information, and the t-1 frame timestamp information.

[0120] S4: Template matching the candidate region to be matched with the search region and calculating a matching position deviation.

[0121] Specifically, the initial movement distance calculated by step S3 and the determined candidate region can obtain the initial estimated position of the candidate region in the t frame. That is, a region of a fixed size is expanded around the initial estimated position as a search region, the candidate region and the search region are template matched, and a matching position deviation is calculated.

[0122] S5: Correcting and compensating the initial movement distance by using the matching position deviation to obtain an inter-frame matching result.

[0123] In the specific embodiment described above, the inter-frame matching detection method based on the binocular vision system provided by the present application respectively acquires the top-view gray image and the top-view segmentation image of the two adjacent frames before and after the same road scene, detects the candidate region to be matched of the previous frame through the top-view segmentation image, calculates the initial movement distance between the two frames and the initial estimated position of the next frame through the vehicle speed and the timestamp, and acquires the search region of the next frame based on the initial estimated position of the next frame; template matches the candidate region to be matched with the search region and calculates a matching position deviation, and finally corrects the initial movement distance by using the matching position deviation to obtain an inter-frame matching result. In this way, the method obtains the position deviation by matching the candidate region to be matched with the search region, corrects and compensates the initial movement distance by using the position deviation, so that the registration accuracy between the two adjacent frames is high, the registration accuracy of the data between the adjacent frames in image processing of the auxiliary driving is improved, and more accurate image processing data is provided for the auxiliary driving system.

[0124] In addition to the above method, the present application also provides an inter-frame matching detection system based on a binocular vision system. In a specific embodiment, as shown in Figure 2 the system comprises:

[0125] an image acquisition processing unit 100 for respectively acquiring the top-view gray image and the top-view segmentation image of the two adjacent frames before and after the same road scene;

[0126] a candidate region acquisition unit 200 for detecting the candidate region to be matched of the previous frame through the top-view segmentation image;

[0127] The search area acquisition unit 300 is configured to calculate an initial moving distance between two frames and an initial estimated position of a next frame by vehicle speed and a time stamp, and acquire a search area of the next frame based on the initial estimated position of the next frame.

[0128] The position deviation acquisition unit 400 is configured to perform template matching on the search area and the candidate area to be matched, and calculate a matching position deviation.

[0129] The matching result output unit 500 is configured to correct the initial moving distance by using the matching position deviation to obtain an inter-frame matching result.

[0130] The image acquisition processing unit 100 is specifically configured to:

[0131] acquire left and right views of a same road scene, and process the left and right views to obtain a dense disparity map of the road scene;

[0132] convert image information of a target area into three-dimensional point cloud information in a world coordinate system based on the dense disparity map, and fit a road surface model based on the three-dimensional point cloud information;

[0133] demarcate a target area in the dense disparity map, input an image of the target area into a trained semantic segmentation model, and obtain segmented two-dimensional image information;

[0134] convert a gray image of a detection area to an XOZ projection plane by homographic transformation based on the two-dimensional image information and the three-dimensional point cloud information to generate the overhead gray image, and convert a segmented image of the detection area to the XOZ projection plane to generate the overhead segmented image.

[0135] The conversion of the image information of the target area into the three-dimensional point cloud information in the world coordinate system based on the dense disparity map specifically includes:

[0136] convert an image coordinate system of the dense disparity map into a world coordinate system based on a binocular stereo vision system imaging model and a pinhole imaging model;

[0137] cut the target area in the dense disparity map based on the target area in the real world coordinate system as a reference;

[0138] convert the image information in the target area into the three-dimensional point cloud information according to the following formula:

[0139] Z = B * F / disp

[0140] X = (Img x -cx) * b / disp

[0141] Y = (Img y-cy)*b / disp

[0142] wherein B is the distance between the optical center of the left camera to the optical center of the right camera in the binocular stereo vision imaging system;

[0143] F is the focal length of the camera in the binocular stereo vision imaging system;

[0144] cx and cy are the image coordinates of the principal point of the camera in the binocular stereo vision imaging system;

[0145] Img x and Img y are the image coordinate points in the target area;

[0146] disp is the disparity value of the image point coordinate (Img x , Img y );

[0147] X is the horizontal distance of the three-dimensional point from the camera in the world coordinate system;

[0148] Y is the vertical distance of the three-dimensional point from the camera in the world coordinate system;

[0149] Z is the depth distance of the three-dimensional point from the camera in the world coordinate system.

[0150] wherein the road surface model equation fitted based on the three-dimensional point cloud information is:

[0151] cosα*X + cosβ*Y + cosγ*Z + D = 0

[0152] wherein cosα is the direction cosine of the angle between the road surface normal vector and the x coordinate axis of the world coordinate system;

[0153] cosβ is the direction cosine of the angle between the road surface normal vector and the y coordinate axis of the world coordinate system;

[0154] cosγ is the direction cosine of the angle between the road surface normal vector and the z coordinate axis of the world coordinate system;

[0155] D is the distance from the origin of the world coordinate system to the road surface.

[0156] wherein the homography transformation based on the two-dimensional image information and the three-dimensional point cloud information is completed by using the following formula:

[0157]

[0158] Proj x = X * scale x

[0159] Proj x = Z * scale z

[0160] wherein X, Z are the transverse distance and the depth distance from the camera respectively;

[0161] Img x and Img y are image coordinate points in the detection region;

[0162] Proj x and Proj y are projection plane coordinate points in the detection region;

[0163] H is the homography transformation matrix.

[0164] The candidate region acquisition unit 200 is specifically configured to:

[0165] On the top-down segmentation map of the previous frame, set the pixel value marked as the preset marker to 1, and set other positions to 0, and generate a binary image therefrom;

[0166] Detect a connected domain on the binary image, and expand a region of a preset size around the center position of the lower boundary of the connected domain as a candidate region.

[0167] The search region acquisition unit 300 is specifically configured to:

[0168] Based on the initial movement distance and the candidate region, obtain an initial estimated position of the candidate region in the next frame;

[0169] Expand a region of a preset size around the initial estimated position as a search region.

[0170] In the above specific embodiments, the inter-frame matching detection system based on a binocular vision system provided by the application acquires a top-down grayscale map and a top-down segmentation map of two adjacent frames before and after a same road scene respectively, detects a candidate region to be matched in the previous frame through the top-down segmentation map, calculates an initial movement distance between the two frames and an initial estimated position of the next frame through vehicle speed and a time stamp, and acquires a search region of the next frame based on the initial estimated position of the next frame; performs template matching on the candidate region to be matched and the search region, calculates a matching position deviation, and finally corrects the initial movement distance by using the matching position deviation to obtain an inter-frame matching result. In this way, the method obtains a position deviation through matching of the candidate region to be matched and the search region, corrects and compensates the initial movement distance by using the position deviation, so that the registration accuracy between adjacent frames is high, the registration accuracy of data between adjacent frames in image processing of the auxiliary driving is improved, and more accurate image processing data is provided for the auxiliary driving system.

[0171] The application further provides an intelligent terminal, comprising: a data acquisition device, a processor and a memory.

[0172] The data acquisition device is configured to acquire data; the memory is configured to store one or more program instructions; and the processor is configured to execute the one or more program instructions to perform the method described above.

[0173] Corresponding to the above-mentioned embodiments, the application further provides a computer storage medium, which comprises one or more program instructions. The one or more program instructions are configured to be executed by a binocular camera depth calibration system to perform the method described above.

[0174] In the embodiments of the application, the processor can be an integrated circuit chip with a signal processing capability. The processor can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0175] The disclosed methods, steps and logic block diagrams in the embodiments of the application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can be any conventional processor. The steps of the method disclosed in combination with the embodiments of the application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The processor reads the information in the storage medium and combines the hardware to complete the steps of the above method.

[0176] The storage medium can be a memory, for example, can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0177] The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory.

[0178] The volatile memory can be Random Access Memory (RAM), used as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The below-described embodiments do not imply that the exclusive use of the disclosed RAM is required. Other types of RAM exist.

[0179] The storage media described in the embodiments of the present application is intended to include, but not be limited to, these and any other suitable types of memory.

[0180] Those skilled in the art should be aware that the functions described in the embodiments of the present application can be implemented in combination of hardware and software in one or more of the above examples. When the software is applied, the corresponding functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on the computer readable medium. The computer readable medium includes computer storage medium and communication medium, wherein the communication medium includes any medium that facilitates the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0181] The above detailed description of the present application is further detailed for the purpose of the present application, technical solutions and beneficial effects. It should be understood that the above is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting inter-frame matching based on a binocular vision system, characterized in that, The method comprises: respectively acquiring an overhead gray image and an overhead segmentation image of two adjacent frames before and after a same road scene; detecting a candidate region to be matched of a previous frame through the overhead segmentation image; calculating an initial moving distance between the two frames and an initial estimated position of a next frame through a vehicle speed and a time stamp, and acquiring a search region of the next frame based on the initial estimated position of the next frame; performing template matching on the candidate region to be matched and the search region, and calculating a matching position deviation; correcting the initial moving distance by using the matching position deviation to obtain an inter-frame matching result; wherein the method of respectively acquiring an overhead gray image and an overhead segmentation image of two adjacent frames before and after a same road scene comprises: acquiring left and right views of the same road scene, and processing the left and right views to obtain a dense disparity map of the road scene; based on the dense disparity map, converting image information of a target region into three-dimensional point cloud information in a world coordinate system, and fitting a road surface model based on the three-dimensional point cloud information; defining a target region in the dense disparity map, inputting an image of the target region into a trained semantic segmentation model, and obtaining two-dimensional image information after segmentation; based on the two-dimensional image information and the three-dimensional point cloud information, performing a homography transformation to convert a gray image of a detection region to an XOZ projection plane to generate the overhead gray image, and to convert a segmentation image of the detection region to the XOZ projection plane to generate the overhead segmentation image; wherein the method of converting image information of a target region into three-dimensional point cloud information in a world coordinate system based on the dense disparity map comprises: based on a binocular stereo vision system imaging model and a pinhole imaging model, converting an image coordinate system of the dense disparity map into a world coordinate system; taking the target region in a real world coordinate system as a reference, intercepting the target region in the dense disparity map; based on the following formula, converting image information in the target region into three-dimensional point cloud information: Z=B*F / disp X = (Img x -cx)*b / disp Y = (Img y -cy)*b / disp wherein B is a distance between a left camera optical center and a right camera optical center in a binocular stereo vision imaging system; F is a camera focal length in the binocular stereo vision imaging system; cx and cy are image coordinates of a camera principal point in the binocular stereo vision imaging system; Img x and Img y are image coordinate points within the target region; disp is the disparity value for the image coordinate point (Img x , Img y ) X is a horizontal distance of a three-dimensional point from a camera in a world coordinate system; Y is a vertical distance of the three-dimensional point from the camera in the world coordinate system; Z is a depth distance of the three-dimensional point from the camera in the world coordinate system; wherein a road surface model equation fitted based on the three-dimensional point cloud information is: cosα*X+cosβ*Y+cosγ*Z+D=0 wherein cosα is a direction cosine of an angle between a road surface normal vector and an x coordinate axis of a world coordinate system; cosβ is a direction cosine of an angle between the road surface normal vector and a y coordinate axis of the world coordinate system; cosγ is a direction cosine of an angle between the road surface normal vector and a z coordinate axis of the world coordinate system; D is a distance from an origin of the world coordinate system to a road surface plane; the homography transformation based on the two-dimensional image information and the three-dimensional point cloud information is completed by using the following formula: Proj x = X * scale x Proj x = Z * scale z wherein X and Z are respectively a horizontal distance from a camera and a depth distance from the camera; Img x and Img y are image coordinate points within the detection region; Proj x and Proj y are the projection plane coordinate points within the detection area; H is the homography transformation matrix.

2. The inter-match detection method of claim 1, wherein, The detecting the candidate region to be matched of the previous frame through the top-view segmentation map specifically comprises: The pixel value on the top-view segmentation map of the previous frame is marked as a preset mark pixel value, and is set to 1, and other positions are set to 0, and a binary image is generated accordingly; A connected domain is detected on the binary image, and a region of a preset size is expanded around the center position of the lower boundary of the connected domain as a candidate region.

3. The inter-match detection method of claim 2, wherein, The obtaining the search region of the next frame based on the initial estimated position of the next frame specifically comprises: Based on the initial moving distance and the candidate region, an initial estimated position of the candidate region in the next frame is obtained; A region of a preset size is expanded around the initial estimated position as a search region.

4. A binocular vision system based inter-frame matching detection system for implementing the method according to any one of claims 1 to 3, characterized in that, The system comprises: An image acquisition unit is configured to acquire a top-view grayscale image and a top-view segmentation map of two adjacent frames in a same road scene respectively; A candidate region acquisition unit is configured to detect a candidate region to be matched of a previous frame through the top-view segmentation map; A search region acquisition unit is configured to calculate an initial moving distance between the two frames and an initial estimated position of a next frame through a vehicle speed and a time stamp, and obtain a search region of the next frame based on the initial estimated position of the next frame; A position deviation acquisition unit is configured to perform template matching on the candidate region to be matched and the search region, and calculate a matching position deviation; A matching result output unit is configured to correct the initial moving distance by using the matching position deviation, so as to obtain an inter-frame matching result.

5. A smart terminal, characterized by The intelligent terminal comprises a data acquisition device, a processor and a memory; The data acquisition device is configured to acquire data; the memory is configured to store one or more program instructions; and the processor is configured to execute the one or more program instructions to perform the method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises one or more program instructions for performing the method according to any one of claims 1-3.

Images