Motor vehicle obstacle detection method, device and computer readable storage medium

Abstract

The embodiment of the present application provides a kind of motor vehicle obstacle detection method, device and computer readable storage medium, the method comprises: obtaining original image frame from video image;Original image frame is handled to obtain original correction graph;Original image frame is handled to obtain compensation correction graph, and image compensation model is based on the imaging plane of vehicle-mounted camera, and original correction graph is constructed and obtained according to the minimum principle of gray difference of non-obstacle region pre-calibrated before and after virtual motion of preset virtual motion matrix;The parallax graph of original correction graph and compensation correction graph is calculated, the parallax derivative graph is obtained by derivation to parallax graph, the actual gray mean is calculated, and the reference gray value of each pixel point is calculated according to actual gray mean and parallax derivative graph;And compared actual gray value and corresponding reference gray value, mark out obstacle pixel point and be obstacle.This embodiment can effectively improve the efficiency and accuracy of obstacle detection.

Description

Technical Field

[0001] The present invention relates to the field of motor vehicle assisted driving technology, and more particularly to a motor vehicle obstacle detection method, device, and computer-readable storage medium. Background Art

[0002] Using onboard cameras to detect generalized obstacles (i.e., obstacles of any type and shape) is a particularly important technology in assisted driving systems. Currently, methods for generalized obstacle detection include motion compensation-based generalized obstacle detection, stereo vision-based generalized obstacle detection, structured light or TOF (time of flight)-based generalized obstacle detection, and deep learning-based obstacle detection.

[0003] However, motion compensation requires accurate motion estimation of the on-board camera. The accuracy of camera motion estimation is poor in conditions such as weak texture areas, structured repetitions, or poor imaging lighting conditions in the image; the stereoscopic vision effect is greatly affected by the manufacturing process, and when multiple cameras are used for obstacle detection, the baseline length is limited due to the spatial arrangement between the lenses, resulting in a small detectable range; the image sensor manufacturing process of the on-board camera based on structured light or TOF is complex and the cost is high; deep learning requires a complex calibration process, and there is a more serious working domain drift problem in the actual detection process; in addition, the above four methods will have false detections for scenes such as non-obstacles projected on the ground and water, interference from strong moving light sources, and bumpy roads, which ultimately leads to failure of obstacle detection and low detection efficiency. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a method for detecting obstacles in a motor vehicle, which can effectively improve the efficiency and accuracy of obstacle detection.

[0005] A further technical problem to be solved by the embodiments of the present invention is to provide a motor vehicle obstacle detection device that can effectively improve the efficiency and accuracy of obstacle detection.

[0006] A further technical problem to be solved by embodiments of the present invention is to provide a computer-readable storage medium for storing a computer program that can effectively improve the efficiency and accuracy of obstacle detection.

[0007] In order to solve the above technical problems, the embodiment of the present invention first provides the following technical solution: a method for detecting obstacles in a motor vehicle, comprising the following steps:

[0008] Extracting original image frames frame by frame from the video images captured by the vehicle-mounted camera;

[0009] Correcting the original image frame according to a pre-stored image correction model to obtain an original corrected image;

[0010] The original image frame is compensated and corrected based on a pre-stored image compensation model and the image correction model to obtain a compensated correction image, wherein the image compensation model is constructed based on the principle of minimizing the grayscale difference of the non-obstacle area pre-calibrated in the original correction image before and after the imaging plane of the vehicle-mounted camera performs virtual motion on the road surface according to a preset virtual motion matrix;

[0011] calculating a disparity map between the original corrected image and the compensated corrected image based on a normalized correlation principle of relative fields of different images, differentiating the disparity map to obtain a disparity derivative map, calculating an actual grayscale mean value of an area above a vanishing line in the disparity derivative map, and calculating a reference grayscale value of each pixel in the disparity derivative map based on the actual grayscale mean value and an actual distance between each pixel and the vanishing line, wherein the reference grayscale value is inversely proportional to the actual distance and directly proportional to the actual grayscale mean; and

[0012] Compare the actual grayscale value of each pixel in the disparity derivative map with the corresponding reference grayscale value, mark the pixel points whose actual grayscale value is greater than the reference grayscale value as obstacle pixels in the disparity derivative map, and mark the area formed by the combination of adjacent obstacle pixels as an obstacle.

[0013] Furthermore, the image correction model is a polyhedron projection correction model that corrects the original image frame into a plurality of the original correction images, the field of view formed by the combination of the plurality of the original correction images is the same as the field of view of the corresponding original image frame, and the original image frame is processed based on the pre-stored image compensation model and the image correction model to obtain a field of view that is consistent with the original correction image. Figure 1 A corresponding compensation correction image, each of the original correction image and the corresponding compensation correction image is calculated based on the normalized correlation principle of the relative fields of different images to generate a disparity derivative image, and the obstacle pixels and obstacles are marked correspondingly for each disparity derivative image.

[0014] Furthermore, the reference gray value ,in, represents the reference grayscale value of each pixel point x in the disparity derivative map, C represents the corresponding normalized correlation within the first predetermined size neighborhood in the disparity map, and D represents the actual distance between the pixel point x and the vanishing line. It represents the actual grayscale mean of the area above the vanishing line in the disparity derivative map.

[0015] Furthermore, the calculating of the disparity map between the original correction image and the compensated correction image based on the normalized correlation principle of relative fields of different images specifically includes:

[0016] Selecting a pixel point in the original corrected image as a first pixel point, calculating a first grayscale mean value within a neighborhood of a second predetermined size with the first pixel point as the center, selecting a pixel point in the compensated corrected image as a second pixel point, calculating a second grayscale mean value within a neighborhood of a second predetermined size with the second pixel point as the center, wherein the second pixel point has the same vertical image coordinates as the second pixel point but has a horizontal image coordinate relative to the first pixel point by d pixels;

[0017] calculating an optimal d value when the normalized correlation between the original corrected image and the compensated corrected image is maximized based on the first grayscale mean, the second grayscale mean, and a pre-stored calculation formula for the normalized correlation between the original corrected image and the compensated corrected image, and using the optimal d value as the grayscale value of a disparity pixel point in the disparity map, where the image coordinates of each disparity pixel point correspond one-to-one to the image coordinates of each first pixel point; and

[0018] The disparity map is obtained by traversing all first pixels in the original corrected image that have corresponding second pixels in the compensated corrected image and corresponding second pixels in the compensated corrected image to calculate the grayscale values ​​corresponding to the disparity pixels.

[0019] Furthermore, the method further comprises the following steps:

[0020] After marking obstacles in the disparity derivative map, the image compensation model is modified according to the obstacles.

[0021] Furthermore, after marking the obstacle in the disparity derivative map, correcting the image compensation model according to the obstacle specifically includes:

[0022] determining whether the obstacle satisfies a preset calibration condition, wherein the preset calibration condition includes that the obstacle is a slender obstacle that meets a predetermined size threshold and the obstacle is within the non-obstacle area in the disparity derivative map;

[0023] When the obstacle meets the preset calibration condition, a correlation mapping between the current frame and the previous frame is calculated based on the LK optical flow of the current frame and the previous frame in the video image;

[0024] calculating a compensated image of the previous frame based on the correlation map;

[0025] Calculating a normalized correlation between the compensated image and the corresponding non-obstacle area in the current frame, and comparing the actual normalized correlation with a preset correlation threshold;

[0026] When the actual normalized correlation is less than the preset correlation threshold, calculating the actual matching area between the current frame and the previous frame that satisfies the preset virtual motion matrix, and using the actual matching area in the current frame to update the non-obstacle area; and

[0027] The image compensation model is modified according to the latest non-obstacle area in the current frame.

[0028] On the other hand, in order to solve the above technical problems, an embodiment of the present invention further provides the following technical solution: a motor vehicle obstacle detection device is connected to a vehicle-mounted camera for shooting and providing video images, the motor vehicle obstacle detection device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the motor vehicle obstacle detection method described in any one of the above items.

[0029] Furthermore, the vehicle-mounted camera is a fisheye camera.

[0030] On the other hand, in order to solve the above technical problems, an embodiment of the present invention further provides the following technical solution: a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle obstacle detection method as described in any one of the above items.

[0031] After adopting the above technical solution, the embodiment of the present invention has at least the following beneficial effects: after the embodiment of the present invention obtains the original image frame from the vehicle-mounted camera, the original image frame is processed to obtain the original correction image, and the image compensation model and the image correction model are used to process the original image frame to obtain the compensation correction image. Since the image compensation model is based on the imaging plane of the vehicle-mounted camera according to the preset virtual motion matrix before and after the virtual movement on the driving road, the grayscale difference of the pre-calibrated non-obstacle area in the original correction image is constructed and obtained. The generation of the compensation correction image is not affected by camera exposure, external lighting, and digital signal conversion, and the compensation correction image and the original correction image meet the photometric consistency assumption. Therefore, the compensation correction image effectively solves the problems that affect the accuracy of motion estimation, such as weak texture, repeated structure and variable scene lighting; since the image compensation model is based on the imaging plane of the vehicle-mounted camera according to the preset virtual motion matrix before and after the virtual movement on the driving road, the generation of the compensation correction image is not affected by camera exposure, external lighting, and digital signal conversion, and the compensation correction image and the original correction image meet the photometric consistency assumption. Therefore, the compensation correction image effectively solves the problems that affect the accuracy of motion estimation, such as weak texture, repeated structure and variable scene lighting; Therefore, the compensated correction image uses the road surface as a priori condition. During the actual detection process, targets such as pseudo obstacles projected into the non-obstacle area, i.e., the road surface, projection interference from a strong moving light source on the road surface, or shallow potholes on the road surface are all classified as targets that meet the requirements of planar motion and are not falsely detected, effectively reducing false detections. Then, after calculating the disparity map between the original correction image and the compensated correction image, since there is a clear trough between the disparity of the obstacle area and the non-obstacle area, the disparity derivative map and the actual grayscale mean of the area above the vanishing line in the disparity derivative map are calculated based on the disparity map. Finally, by comparing the actual grayscale value of each pixel in the disparity derivative map with the corresponding reference grayscale value, the obstacle pixel in the disparity derivative map can be determined, and the area formed by combining adjacent obstacle pixels can be marked as an obstacle, which can effectively improve the efficiency and accuracy of obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flowchart of an optional embodiment of the motor vehicle obstacle detection method of the present invention.

[0033] Figure 2 This is a specific flow chart of step S4 of an optional embodiment of the motor vehicle obstacle detection method of the present invention.

[0034] Figure 3 This is a flowchart of another optional embodiment of the motor vehicle obstacle detection method of the present invention.

[0035] Figure 4 This is a specific flow chart of step S6 of an optional embodiment of the motor vehicle obstacle detection method of the present invention.

[0036] Figure 5 The figure is a principle block diagram of an optional embodiment of the motor vehicle obstacle detection device of the present invention.

[0037] Figure 6 This is a functional module diagram of an optional embodiment of the motor vehicle obstacle detection device of the present invention. DETAILED DESCRIPTION

[0038] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the following exemplary embodiments and descriptions are only used to explain the present invention and are not intended to limit the present invention. Moreover, the embodiments and features in the embodiments of the present application may be combined with each other unless there is a conflict.

[0039] like Figure 1 As shown, an optional embodiment of the present invention provides a method for detecting obstacles in a motor vehicle, comprising the following steps:

[0040] S1: Extracting original image frames frame by frame from the video images captured by the vehicle-mounted camera 1;

[0041] S2: performing correction processing on the original image frame according to a pre-stored image correction model to obtain an original corrected image;

[0042] S3: Compensating and correcting the original image frame based on a pre-stored image compensation model and the image correction model to obtain a compensated correction image, wherein the image compensation model is constructed based on the principle of minimizing the grayscale difference of the non-obstacle area pre-calibrated in the original correction image before and after the imaging plane of the vehicle-mounted camera 1 performs virtual motion on the road surface according to a preset virtual motion matrix;

[0043] S4: calculating a disparity map between the original corrected image and the compensated corrected image based on the normalized correlation principle of relative fields of different images, differentiating the disparity map to obtain a disparity derivative map, calculating an actual grayscale mean value of an area above a vanishing line in the disparity derivative map, and calculating a reference grayscale value of each pixel in the disparity derivative map based on the actual grayscale mean value and an actual distance between each pixel and the vanishing line, wherein the reference grayscale value is inversely proportional to the actual distance and directly proportional to the actual grayscale mean; and

[0044] S5: Compare the actual grayscale value of each pixel in the disparity derivative map with the corresponding reference grayscale value, mark the pixel points whose actual grayscale value is greater than the reference grayscale value as obstacle pixels in the disparity derivative map, and mark the area formed by the combination of adjacent obstacle pixels as an obstacle.

[0045] The embodiment of the present invention obtains the original image frame from the vehicle-mounted camera 1, processes the original image frame to obtain the original correction image, and simultaneously uses the image compensation model and the image correction model to process the original image frame to obtain the compensation correction image. Since the image compensation model is based on the principle of minimum grayscale difference of the non-obstacle area pre-calibrated in the original correction image before and after the imaging plane of the vehicle-mounted camera 1 performs virtual movement on the driving road according to the preset virtual motion matrix, the generation of the compensation correction image is not affected by camera exposure, external lighting, and digital signal conversion, and the compensation correction image and the original correction image meet the photometric consistency assumption. Therefore, the compensation correction image effectively solves the problems that affect the accuracy of motion estimation, such as weak texture, repeated structure, and variable scene lighting; since the image compensation model is based on the imaging plane of the vehicle-mounted camera 1 performs virtual movement on the driving road according to the preset virtual motion matrix, the compensation correction image is not affected by camera exposure, external lighting, and digital signal conversion, and the compensation correction image and the original correction image meet the photometric consistency assumption. Therefore, the compensation correction image effectively solves the problems that affect the accuracy of motion estimation, such as weak texture, repeated structure, and variable scene lighting; The compensated correction image uses the road surface as a priori. During actual detection, objects projected onto the non-obstacle area (i.e., the road surface), interference from strong moving light sources, or shallow potholes on the road surface are all classified as targets that meet the requirements of planar motion and are not falsely detected, effectively reducing false detections. Then, after calculating the disparity map between the original correction image and the compensated correction image, a disparity derivative map and the actual grayscale mean of the area above the vanishing line in the disparity derivative map are calculated based on the disparity map. Finally, by comparing the actual grayscale value of each pixel in the disparity derivative map with the corresponding reference grayscale value, the obstacle pixels in the disparity derivative map are identified. The area formed by combining adjacent obstacle pixels is then marked as an obstacle, effectively improving the efficiency and accuracy of obstacle detection. In specific implementations, steps S2 and S3 can be performed in any order and can be performed simultaneously.

[0046] In an optional embodiment of the present invention, the image correction model is a polyhedron projection correction model that corrects the original image frame into a plurality of the original correction images, the field of view formed by the combination of the plurality of the original correction images is the same as the field of view of the corresponding original image frame, and the original image frame is processed based on the pre-stored image compensation model and the image correction model to obtain a field of view that is consistent with the original correction image. Figure 1A corresponding compensated correction image is generated. Each original correction image and the corresponding compensated correction image are calculated based on the normalized correlation principle of the relative fields of different images to generate a disparity derivative map. Obstacle pixels and obstacles are correspondingly marked for each disparity derivative map. In this embodiment, the image correction model utilizes a polyhedron projection correction model. The polyhedron projection correction model processes a single original image frame to generate multiple original correction images captured by multiple narrow-field-of-view cameras. By performing obstacle detection on each original correction image separately, obstacle detection can be achieved across a wide field of view of the original image frame.

[0047] In specific implementation, since the angle between any point P in the imaging plane A of the vehicle-mounted camera 1 and the optical center O is less than 90 degrees, the field of view of the image captured by a wide-field lens (e.g., a fisheye lens) with an incident angle θ greater than 90 degrees will be lost after conventional image correction. The polyhedron projection correction model rotates the original imaging plane perpendicular to the optical axis (in an ideal state) by a predetermined angle α around an axis parallel to the imaging plane to obtain a larger incident angle and achieve polyhedron fisheye full-field correction. The specific process of constructing the polyhedron projection correction model is as follows:

[0048] In step S2, first, a three-dimensional space rotation matrix is ​​constructed for rotating the imaging plane, where the rotation matrix is ​​defined as R:

[0049] (Formula 1)

[0050] Then, the normalized imaging plane point P is set to be expressed as:

[0051] (Formula 2)

[0052] Next, the imaging plane point P is rotated to the specified plane Pr as follows:

[0053] (Formula 3)

[0054] Furthermore, the theoretical incident angle θ after rotation is expressed as:

[0055] (Formula 4)

[0056] Furthermore, the imaging radius of the original image frame under the theoretical incident angle θ is calculated. Expressed as:

[0057] (Formula 5)

[0058] in, Represents the polynomial coefficients of the distortion curve of the vehicle-mounted camera;

[0059] Furthermore, the correspondence between the image coordinates (u, v) of the rotated imaging plane point Pr and the original image frame is calculated as follows:

[0060] (Formula 6)

[0061] Among them, (Xc, Zc) represents the center of the normalized imaging plane, ( , ) represents the image center of the original image frame;

[0062] Furthermore, the corresponding relationship between the image coordinates of the rotated imaging plane point Pr and the original correction image is calculated as follows:

[0063] (Formula 7)

[0064] in( , ) is the image coordinate of the original correction image corresponding to the imaging plane point Pr after rotation, and Represent the width and height of the original rectified image respectively;

[0065] Finally, combining Formula 1-Formula 7, the polyhedron projection correction model can be expressed as:

[0066] (Formula 8)

[0067] in, and represent the original correction image and the original image frame respectively;

[0068] In a specific implementation, by adjusting the value of the predetermined angle α, multiple original correction images can be obtained based on an original image frame, and each original correction image corresponds to a value of the predetermined angle α. In a specific embodiment of the embodiment of the present invention, the predetermined angle α is usually set to three values, and three original correction images are generated accordingly.

[0069] In step S3, to highlight the difference between the 3D obstacle and its background (non-3D obstacle area), the embodiment of the present invention first sets the vector of the preset camera virtual motion to meet the following conditions: rotation around an axis parallel to the imaging plane of the vehicle-mounted camera and translation in a plane perpendicular to the imaging plane (for example, a vehicle moving on a road). The preset virtual motion matrix is ​​expressed as:

[0070] (Formula 9)

[0071] in, Indicates the angle of rotation of the preset vehicle camera around the axis perpendicular to the road surface. and Indicates the lateral and longitudinal distances of the simulated vehicle-mounted camera moving parallel to the road surface, both of which are preset empirical values;

[0072] Furthermore, the point on the imaging plane after the virtual motion of the vehicle-mounted camera is calculated first, and the point Pr in Formula 3 is translated and rotated based on the camera virtual motion matrix to obtain the point in the imaging plane after the virtual motion:

[0073] (Formula 10)

[0074] Further, the point Substitute Pr into formula 3 and perform image correction according to formula 4-8 to obtain the virtual motion image after the original correction image is subjected to virtual motion. ;

[0075] Furthermore, the original rectified image The pre-calibrated non-obstacle area (i.e., the driving road) is expressed as:

[0076] (Formula 11)

[0077] Furthermore, based on virtual motion images and the original rectified image The non-obstacle area in is used to estimate the camera motion based on the direct method:

[0078] First, the correspondence between the imaging plane points of the front and rear frames of the vehicle-mounted camera is expressed as:

[0079] (Formula 12)

[0080] in, is the plane point after movement, A Lie algebra representing the preset virtual motion matrix;

[0081] Next, calculate the projection coordinates of the image after the vehicle camera moves ( , ) can be expressed as:

[0082] (Formula 13)

[0083] in, and are the focal lengths of the corrected image in the U and V directions of the ideal lens respectively; (cx, cy) is the center of the corrected image.

[0084] Next, based on the principle of minimizing the grayscale difference of the pre-calibrated non-obstacle area in the original correction image before and after the virtual movement of the imaging plane of the vehicle-mounted camera 1 according to the preset virtual motion matrix on the road surface, the motion of the vehicle-mounted camera 1 is estimated:

[0085] (Formula 14)

[0086] in, represents the non-obstacle area of ​​the original correction image, Indicates the non-obstacle area corresponding to the image after the original correction image undergoes virtual movement;

[0087] Then, based on formula 14, we can calculate Compensated virtual motion map We can get:

[0088] (Formula 15)

[0089] Finally, point Substitute Pr into formula 3 and perform image correction according to formula 4-8 to obtain the compensation correction map .

[0090] In an optional embodiment of the present invention, the reference gray value ,in, represents the reference grayscale value of each pixel point x in the disparity derivative map, C represents the corresponding normalized correlation within the first predetermined size neighborhood in the disparity map, and D represents the actual distance between the pixel point x and the vanishing line. In this embodiment, the reference gray value is calculated using the formula as above: , and introduce the normalized correlation within the first predetermined size neighborhood in the disparity map into the calculation of the reference grayscale value The normalized correlation within the first predetermined size neighborhood in the disparity map can be used to evaluate the texture richness within the first predetermined size neighborhood of the disparity map, thereby improving the accuracy of subsequent obstacle detection. In specific implementations, the first predetermined size neighborhood is typically set to a 32*32 square area.

[0091] In an optional embodiment of the present invention, Figure 2 As shown, the step S4 specifically includes:

[0092] S41: selecting a pixel point in the original corrected image as a first pixel point, calculating a first grayscale mean value within a neighborhood of a second predetermined size with the first pixel point as the center, selecting a pixel point in the compensated corrected image as a second pixel point, and calculating a second grayscale mean value within a neighborhood of a second predetermined size with the second pixel point as the center, wherein the second pixel point has the same vertical image coordinates as the second pixel point but has a horizontal image coordinate relative to the first pixel point by d pixels;

[0093] S42: calculating an optimal d value when the normalized correlation between the original corrected image and the compensated corrected image is maximum according to the first grayscale mean, the second grayscale mean, and a pre-stored calculation formula for the normalized correlation between the original corrected image and the compensated corrected image, and using the optimal d value as the grayscale value of a disparity pixel point in the disparity image, wherein the image coordinates of each of the disparity pixel points correspond one-to-one to the image coordinates of each of the first pixel points; and

[0094] S43: traverse all first pixels in the original corrected image that have corresponding second pixels in the compensated corrected image and corresponding second pixels in the compensated corrected image to calculate the grayscale values ​​corresponding to the disparity pixels to obtain the disparity map.

[0095] In this embodiment, theoretically, the compensated correction image is an image of the original correction image after calculating the compensation for camera motion in the non-obstacle area. Therefore, the grayscale difference between the compensated correction image and the original correction image in the obstacle area is much greater than that in the non-obstacle area. The present invention proposes to use the disparity between the compensated correction image and the original correction image to detect obstacles. There is a clear trough between the disparity in the obstacle area and the disparity in the non-obstacle area, which helps to extract obstacles.

[0096] In step S41, the original correction image Take a point x and calculate the first grayscale mean value on the second predetermined size area R with x as the center , similarly, in the compensation correction diagram Calculate the second grayscale mean value on a second predetermined size area R with x+d as the center Typically, the second predetermined size area R is a circular area of ​​32*32.

[0097] In step S42, the calculation formula based on the normalized correlation between the original correction image and the compensated correction image is:

[0098] (Formula 16)

[0099] Based on the NCC calculation formula, the local correlation between the two original correction images and the compensation correction image is calculated. The virtual epipolar line (because the camera is in virtual motion, the ideal motion vector is set and no epipolar correction is required) traverses an area near the field, which is consistent with the original correction map The NCC value is the largest in the area defined in the calculation. At this time, the d value is the difference between the original correction image and the compensation correction image. Figure 1 The parallax of point x;

[0100] In step S43, the original correction image Each point in the image is traversed and executed through steps S41 and S42 to obtain a disparity map that is consistent with the image size. ;

[0101] Next, the disparity map Derivative to obtain the parallax derivative map ;

[0102] Furthermore, the disparity derivative map is calculated The actual grayscale mean of the area above the vanishing line ;

[0103] Further, calculate the reference gray value (Formula 17)

[0104] in, Represents the reference grayscale value of the coordinate point x in the disparity derivative map.

[0105] Finally, in step S5, the obstacle pixels are marked and the obstacle image is output according to the following formula: :

[0106] (Equation 18).

[0107] In an optional embodiment of the present invention, Figure 3 As shown, the method further includes the following steps:

[0108] S6: After marking the obstacle in the disparity derivative map, the image compensation model is modified according to the obstacle.

[0109] In this embodiment, the image compensation model is modified according to the detected obstacle, thereby improving the accuracy of subsequent obstacle detection.

[0110] In an optional embodiment of the present invention, Figure 4 As shown, the step S6 specifically includes:

[0111] S61: Determine whether the obstacle meets a preset calibration condition, where the preset calibration condition includes that the obstacle is a slender obstacle that meets a predetermined size threshold and the obstacle is within the non-obstacle area in the disparity derivative map;

[0112] S62: When the obstacle meets the preset calibration condition, a correlation mapping between the current frame and the previous frame is calculated based on the LK optical flow of the current frame and the previous frame in the video image;

[0113] S63: Calculating the compensated image of the previous frame based on the correlation map;

[0114] S64: Calculating a normalized correlation between the compensated image and the corresponding non-obstacle area in the current frame, and comparing the actual normalized correlation with a preset correlation threshold;

[0115] S65: when the actual normalized correlation is less than the preset correlation threshold, calculating the actual matching area between the current frame and the previous frame that satisfies the preset virtual motion matrix, and using the actual matching area in the current frame to update the non-obstacle area; and

[0116] S66: Modify the image compensation model according to the latest non-obstacle area in the current frame.

[0117] In combination with the above, in the embodiment of the present invention, the non-obstacle area in the original correction image is manually calibrated when the obstacle detection system is initially run for the first frame of the video image. In subsequent frames, obstacles are usually slender, which is inconsistent with normal experience, and obstacles should not be in the non-obstacle area. Therefore, when the above-mentioned preset calibration conditions are met, the system determines that there is an error in obstacle detection and needs to recalibrate the non-obstacle area; further, based on the LK optical flow and correlation mapping principles, a compensation image of the current frame of the video image is calculated in sequence, without considering the state of motion mutation. The normalized correlation between the compensation image and the current frame under normal conditions should be high. By judging the correlation between the compensation image and the current frame, the compensation image is calculated. The normalized correlation of the non-obstacle area corresponding to the current frame is further determined to further determine that the non-obstacle area cannot meet the current movement of the motor vehicle. Since the on-board camera performs virtual movement on the driving road according to the preset virtual motion matrix, which strictly simulates the driving road, the points in the non-obstacle area of ​​two adjacent frames in the video image are closer to the simulated movement than the points in the three-dimensional obstacle area. Therefore, the actual matching area that meets the preset virtual motion matrix in the current frame and the previous frame is calculated, and the actual matching area in the current frame is used to update the non-obstacle area. Finally, the image compensation model can be corrected based on the latest non-obstacle area in the current frame.

[0118] On the other hand, Figure 5As shown, an embodiment of the present invention further provides a motor vehicle obstacle detection device 3, which is connected to a vehicle-mounted camera 1 for shooting and providing video images. The motor vehicle obstacle detection device 3 includes a processor 30, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor. When the processor 30 executes the computer program, it implements the motor vehicle obstacle detection method as described in any one of the above items.

[0119] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments that can implement specific functions, and the instruction segments are used to describe the execution process of the computer program in the motor vehicle obstacle detection device 3. For example, the computer program may be divided into Figure 6 The functional modules in the motor vehicle obstacle detection device 3, wherein the image frame extraction module 41, the original correction image calculation module 42, the compensation correction image calculation module 43, the disparity derivative map calculation module 44 and the obstacle marking module 45 respectively execute the above steps S1 to S5.

[0120] The vehicle obstacle detection device 3 can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The vehicle obstacle detection device 3 can include, but is not limited to, a processor 30 and a memory 32. Those skilled in the art will appreciate that the schematic diagram is merely an example of the vehicle obstacle detection device 3 and does not limit the vehicle obstacle detection device 3. The vehicle obstacle detection device 3 may include more or fewer components than shown, or a combination of certain components, or different components. For example, the vehicle obstacle detection device 3 may also include input and output devices, network access devices, buses, and the like.

[0121] The processor 30 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor 30 serves as the control center of the vehicle obstacle detection device 3 and connects all components of the vehicle obstacle detection device 3 using various interfaces and circuits.

[0122] The memory 32 can be used to store the computer programs and / or modules. The processor 30 implements the various functions of the vehicle obstacle detection device 3 by running or executing the computer programs and / or modules stored in the memory 32 and accessing the data stored in the memory 32. The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as pattern recognition and pattern overlay functions); the data storage area may store data (such as graphic data) generated during the use of the vehicle obstacle detection device 3. Furthermore, the memory 32 may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0123] If the functions described in the embodiments of the present invention are implemented in the form of software modules or units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by the processor 30, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0124] In an optional embodiment of the present invention, the vehicle-mounted camera 1 is a fisheye camera. In this embodiment, the vehicle-mounted camera 1 adopts a fisheye camera, which captures images with a wide field of view and can provide a larger field of view for obstacle detection for the vehicle.

[0125] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle obstacle detection method as described in any one of the above items.

[0126] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0127] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which fall within the scope of protection of the present invention.

Claims

1. A method for detecting obstacles in a motor vehicle, characterized in that: The method comprises the following steps: Extracting original image frames frame by frame from the video images captured by the vehicle-mounted camera; Correcting the original image frame according to a pre-stored image correction model to obtain an original corrected image; The original image frame is compensated and corrected based on a pre-stored image compensation model and the image correction model to obtain a compensated correction image, wherein the image compensation model is constructed based on the principle of minimizing the grayscale difference of the non-obstacle area pre-calibrated in the original correction image before and after the imaging plane of the vehicle-mounted camera performs virtual motion on the road surface according to a preset virtual motion matrix; calculating a disparity map between the original corrected image and the compensated corrected image based on a normalized correlation principle of relative fields of different images, differentiating the disparity map to obtain a disparity derivative map, calculating an actual grayscale mean value of an area above a vanishing line in the disparity derivative map, and calculating a reference grayscale value of each pixel in the disparity derivative map based on the actual grayscale mean value and an actual distance between each pixel and the vanishing line, wherein the reference grayscale value is inversely proportional to the actual distance and directly proportional to the actual grayscale mean; and Compare the actual grayscale value of each pixel in the disparity derivative map with the corresponding reference grayscale value, mark the pixel points whose actual grayscale value is greater than the reference grayscale value as obstacle pixels in the disparity derivative map, and mark the area formed by the combination of adjacent obstacle pixels as an obstacle.

2. The motor vehicle obstacle detection method according to claim 1, wherein: The image correction model is a polyhedron projection correction model that corrects the original image frame into multiple original correction images. The field of view formed by the combination of the multiple original correction images is the same as the field of view of the corresponding original image frame. The original image frame is processed based on the pre-stored image compensation model and the image correction model to obtain a compensated correction image that corresponds one-to-one to the original correction image. Each original correction image and the corresponding compensated correction image are calculated based on the normalized correlation principle of different image relative fields to generate a disparity derivative map. The obstacle pixel points and obstacles are marked accordingly for each disparity derivative map.

3. The motor vehicle obstacle detection method according to claim 1, wherein: The reference gray value ,in, represents the reference grayscale value of each pixel point x in the disparity derivative map, C represents the corresponding normalized correlation within the first predetermined size neighborhood in the disparity map, and D represents the actual distance between the pixel point x and the vanishing line. It represents the actual grayscale mean of the area above the vanishing line in the disparity derivative map.

4. The motor vehicle obstacle detection method according to claim 1, wherein: The step of calculating the disparity map between the original correction image and the compensated correction image based on the normalized correlation principle of relative fields of different images specifically includes: Selecting a pixel point in the original corrected image as a first pixel point, calculating a first grayscale mean value within a neighborhood of a second predetermined size with the first pixel point as the center, selecting a pixel point in the compensated corrected image as a second pixel point, calculating a second grayscale mean value within a neighborhood of a second predetermined size with the second pixel point as the center, wherein the second pixel point has the same vertical image coordinates as the second pixel point but has a horizontal image coordinate relative to the first pixel point by d pixels; calculating an optimal d value when the normalized correlation between the original corrected image and the compensated corrected image is maximized based on the first grayscale mean, the second grayscale mean, and a pre-stored calculation formula for the normalized correlation between the original corrected image and the compensated corrected image, and using the optimal d value as the grayscale value of a disparity pixel point in the disparity map, where the image coordinates of each disparity pixel point correspond one-to-one to the image coordinates of each first pixel point; and The disparity map is obtained by traversing all first pixels in the original corrected image that have corresponding second pixels in the compensated corrected image and corresponding second pixels in the compensated corrected image to calculate the grayscale values ​​corresponding to the disparity pixels.

5. The motor vehicle obstacle detection method according to claim 1, wherein: The method further comprises the following steps: After marking obstacles in the disparity derivative map, the image compensation model is modified according to the obstacles.

6. The motor vehicle obstacle detection method according to claim 5, characterized in that: After marking the obstacle in the disparity derivative map, correcting the image compensation model according to the obstacle specifically includes: determining whether the obstacle satisfies a preset calibration condition, wherein the preset calibration condition includes that the obstacle is a slender obstacle that meets a predetermined size threshold and the obstacle is within the non-obstacle area in the disparity derivative map; When the obstacle meets the preset calibration condition, a correlation mapping between the current frame and the previous frame is calculated based on the LK optical flow of the current frame and the previous frame in the video image; calculating a compensated image of the previous frame based on the correlation map; Calculating a normalized correlation between the compensated image and the corresponding non-obstacle area in the current frame, and comparing the actual normalized correlation with a preset correlation threshold; When the actual normalized correlation is less than the preset correlation threshold, calculating the actual matching area between the current frame and the previous frame that satisfies the preset virtual motion matrix, and using the actual matching area in the current frame to update the non-obstacle area; and The image compensation model is modified according to the latest non-obstacle area in the current frame.

7. A motor vehicle obstacle detection device connected to a vehicle-mounted camera for capturing and providing video images, characterized in that: The motor vehicle obstacle detection device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the motor vehicle obstacle detection method according to any one of claims 1 to 6 is implemented.

8. The motor vehicle obstacle detection device according to claim 7, wherein: The vehicle-mounted camera is a fisheye camera.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle obstacle detection method according to any one of claims 1 to 6.

Images