Motor vehicle three-dimensional obstacle detection and reconstruction method and device and computer readable storage medium

Abstract

The embodiment of the invention provides a motor vehicle three-dimensional obstacle detection and reconstruction method and device and a computer readable storage medium. The method comprises the following steps: extracting an original image frame from a video image of a vehicle-mounted camera; determining two-dimensional initial information of an obstacle and the type of the obstacle; performing image processing on the original image frame to respectively obtain a fitting ground plane and an aerial view corresponding to the original image frame, and calculating three-dimensional initial information of an obstacle contained in the original image frame; calculating a first center point coordinate of each obstacle detected by a vehicle-mounted radar based on data of obstacles around the motor vehicle provided by the vehicle-mounted radar; screening and fusing the two-dimensional initial information, the three-dimensional initial information and the first center point coordinate corresponding to the same obstacle to obtain three-dimensional target information; and rendering the obstacle to a pre-constructed three-dimensional scene map of the motor vehicle. According to the embodiment, the detection precision of the three-dimensional obstacle can be effectively improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of motor vehicle assisted driving technology, and in particular, to a motor vehicle three-dimensional obstacle detection and reconstruction method, device and computer-readable storage medium. Background Art

[0002] When implementing intelligent assisted driving, the motor vehicle assisted driving system will rationally plan the driving route based on the three-dimensional scene map obtained in advance through three-dimensional obstacle detection and reconstruction to avoid collisions between the motor vehicle and obstacles.

[0003] An existing three-dimensional obstacle detection and reconstruction method is to directly convert the original image taken by the vehicle-mounted camera into a bird's-eye view image, and then detect and reconstruct the three-dimensional obstacle based on the bird's-eye view.

[0004] However, the inventors have found in specific implementations that the above method only performs image processing through visual images, and it is usually difficult to accurately know the distance between the motor vehicle and the obstacle, resulting in relatively poor detection accuracy of three-dimensional obstacles. Summary of the invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a method for detecting and reconstructing three-dimensional obstacles of a motor vehicle, which can effectively improve the detection accuracy of three-dimensional obstacles.

[0006] A further technical problem to be solved by the embodiments of the present invention is to provide a three-dimensional obstacle detection and reconstruction device for a motor vehicle, which can effectively improve the detection accuracy of three-dimensional obstacles.

[0007] A further technical problem to be solved by the embodiments of the present invention is to provide a computer-readable storage medium for storing a computer program that can effectively improve the detection accuracy of three-dimensional obstacles.

[0008] In order to solve the above technical problems, the embodiment of the present invention provides the following technical solutions: a method for detecting and reconstructing three-dimensional obstacles of a motor vehicle, comprising the following steps: Extracting original image frames from video images of the surrounding environment of the motor vehicle provided by each on-board camera of the motor vehicle; Determining two-dimensional initial information of an obstacle contained in the original image frame and the type of the obstacle; Performing image processing on the original image frames in combination with the pre-calibrated camera poses of each of the vehicle-mounted cameras to obtain a fitted ground plane and a bird's-eye view corresponding to the original image frames, respectively, and calculating three-dimensional initial information of obstacles contained in the original image frames in combination with the fitted ground plane and the bird's-eye view; Calculate the first center point coordinates of each obstacle detected by the vehicle-mounted radar based on the obstacle data around the motor vehicle provided by the vehicle-mounted radar detection; Filtering and fusing the two-dimensional initial information, the three-dimensional initial information, and the first center point coordinates corresponding to the same obstacle to obtain three-dimensional target information of each obstacle; and Based on the type and three-dimensional target information of the obstacle, the obstacle is rendered into a pre-constructed three-dimensional scene map of the motor vehicle.

[0009] Furthermore, the determining of the two-dimensional initial information of the obstacle contained in the original image frame specifically includes: Converting the original image frame into an original grayscale image; Using the HOG algorithm model to extract each image feature in the original grayscale image; and The SVM algorithm model is combined with the sliding window technology to select obstacles from various image features, and the two-dimensional initial information corresponding to the obstacle and the type of the obstacle are determined.

[0010] Furthermore, the image processing of the original image frame in combination with the pre-calibrated camera poses of each of the vehicle-mounted cameras to obtain the fitted ground plane and the bird's-eye view corresponding to the original image frame, and calculating the three-dimensional initial information of the obstacle contained in the original image frame in combination with the fitted ground plane and the bird's-eye view specifically includes: Converting the original image frame into a bird's-eye view in combination with the pre-calibrated camera poses of each of the vehicle-mounted cameras; Using the PTN network model to process the original image frame to generate corresponding three-dimensional image information; Using a random sampling consensus algorithm to perform ground plane fitting on the three-dimensional image information to generate a fitted ground plane; and Three-dimensional object detection is performed based on the fitted ground plane in combination with the bird's-eye view to obtain three-dimensional initial information of obstacles.

[0011] Furthermore, the original image frame is converted into a bird's-eye view by using a BirGAN network model combined with the pre-calibrated camera poses of each of the vehicle-mounted cameras.

[0012] Furthermore, the step of calculating the first center point coordinates of each obstacle detected by the vehicle-mounted radar based on the obstacle data around the motor vehicle provided by the vehicle-mounted radar detection specifically includes: Using the K-MEANS clustering algorithm model to perform clustering processing on the obstacle data; and The center points of obstacle data in the same category after clustering are calculated in sequence to obtain the first center point coordinates of each obstacle detected by the vehicle-mounted radar.

[0013] Furthermore, the two-dimensional initial information, the three-dimensional initial information and the first center point coordinates corresponding to the same obstacle are screened by the following method: Projecting the first center point coordinates, the second center point coordinates included in the two-dimensional initial information, and the three-dimensional initial information into the bird's-eye view based on the camera poses pre-calibrated based on each vehicle-mounted camera to respectively generate first projection coordinates, second projection coordinates, and a two-dimensional plane projection frame; and Determine whether the first projection coordinates and the second projection coordinates are within the two-dimensional plane projection frame. If the first projection coordinates and the second projection coordinates are both within the two-dimensional plane projection frame, determine that the corresponding first center point coordinates, the two-dimensional initial information and the two-dimensional initial information belong to the same obstacle.

[0014] Furthermore, the coordinates of the first center point are (x l ,y l The two-dimensional initial information includes the length w of the two-dimensional preliminary selection box containing the obstacle. 1 、Width h 1 And the coordinates of the second center point (c xl , c yl The three-dimensional initial information includes the length w of the three-dimensional preliminary selection box containing the obstacle. 2 、Width h 2 , height l, coordinates of the third center point (c x2 , c y2 , c z2 ) and a rotation angle θ; the three-dimensional target information includes the length w of the three-dimensional target frame containing the obstacle x 、Width h x , height l x , target center point coordinates (c x , c y , c z ) and the rotation angle θ x ,in: w x =max(w 1 , w 2 ); h x =max(h 1 ,h 2 ); l x =l; like ,but ,otherwise ; like ,but ,otherwise ; c z = c z2 ; θ x =θ.

[0015] On the other hand, in order to solve the above-mentioned further technical problems, an embodiment of the present invention further provides the following technical solutions: a three-dimensional obstacle detection and reconstruction device for a motor vehicle, connected to a vehicle-mounted camera and a vehicle-mounted radar, the device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, the three-dimensional obstacle detection and reconstruction method for a motor vehicle as described in any one of the above items is implemented.

[0016] On the other hand, in order to solve the above-mentioned further technical problems, an embodiment of the present invention also provides the following technical solutions: a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle three-dimensional obstacle detection and reconstruction method as described in any one of the above-mentioned items.

[0017] After adopting the above technical solution, the embodiment of the present invention has at least the following beneficial effects: after extracting the original image frame from the video image of the surrounding environment of the motor vehicle provided by each vehicle-mounted camera of the motor vehicle, the embodiment of the present invention first determines the two-dimensional initial information of the obstacle contained in the original image frame and the type of the obstacle, and then converts the original image frame into a corresponding bird's-eye view and obtains a fitting ground plane in combination with the camera posture pre-calibrated by the vehicle-mounted camera, and then calculates the three-dimensional initial information of the obstacle contained in the original image frame in combination with the bird's-eye view and the fitting ground plane, and further calculates the first center point coordinates of each obstacle detected by the vehicle-mounted radar based on the obstacle data of the obstacle detected by the vehicle-mounted radar, so that the various types of data corresponding to the same obstacle can be screened and fused to generate the three-dimensional target information of the obstacle, and finally the obstacle is rendered into the three-dimensional scene map of the motor vehicle. Since the three-dimensional target information fuses the two-dimensional initial information and the three-dimensional initial information, the position detection accuracy of the close-range three-dimensional obstacle of the motor vehicle can be effectively improved, and the first center point coordinates obtained by the fused obstacle data detected by the vehicle-mounted radar can further improve the position detection accuracy, reduce the false detection situation, and effectively improve the stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a flowchart of the steps of an optional embodiment of the method for detecting and reconstructing three-dimensional obstacles of a motor vehicle.

[0019] Figure 2 This is a specific flow chart of step S2 of an optional embodiment of the method for detecting and reconstructing three-dimensional obstacles of a motor vehicle according to the present invention.

[0020] Figure 3 This is a specific flow chart of step S3 of an optional embodiment of the method for detecting and reconstructing three-dimensional obstacles of a motor vehicle according to the present invention.

[0021] Figure 4 This is a specific flow chart of step S4 of an optional embodiment of the method for detecting and reconstructing three-dimensional obstacles of a motor vehicle according to the present invention.

[0022] Figure 5 The figure is a principle block diagram of an optional embodiment of the three-dimensional obstacle detection and reconstruction device for motor vehicles of the present invention.

[0023] Figure 6 This is a functional module diagram of an optional embodiment of the motor vehicle three-dimensional obstacle detection and reconstruction device of the present invention. DETAILED DESCRIPTION

[0024] The present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following illustrative embodiments and descriptions are only used to explain the present invention and are not intended to limit the present invention. In addition, the embodiments and features in the embodiments of the present application may be combined with each other without conflict.

[0025] like Figure 1 As shown, an optional embodiment of the present invention provides a method for detecting and reconstructing three-dimensional obstacles of a motor vehicle, comprising the following steps: S1: extracting original image frames from video images of the surrounding environment of the motor vehicle provided by each vehicle-mounted camera 1 of the motor vehicle; S2: Determine the two-dimensional initial information of the obstacle contained in the original image frame and the type of the obstacle; S3: performing image processing on the original image frame in combination with the pre-calibrated camera poses of each of the vehicle-mounted cameras 1 to obtain a fitted ground plane and a bird's-eye view corresponding to the original image frame, and calculating the three-dimensional initial information of the obstacle contained in the original image frame in combination with the fitted ground plane and the bird's-eye view; S4: Calculating the first center point coordinates of each obstacle detected by the vehicle-mounted radar 2 based on the obstacle data around the motor vehicle provided by the vehicle-mounted radar 2; and S5: Filter and fuse the two-dimensional initial information, the three-dimensional initial information and the first center point coordinates corresponding to the same obstacle to obtain the three-dimensional target information of each obstacle S6: Based on the type of the obstacle and the three-dimensional target information, the obstacle is rendered (for example, using OpenGL to implement rendering) into a pre-built three-dimensional scene map of the motor vehicle.

[0026] After extracting the original image frame from the video image of the surrounding environment of the motor vehicle provided by each vehicle-mounted camera 1 of the motor vehicle, the embodiment of the present invention first determines the two-dimensional initial information of the obstacle contained in the original image frame and the type of the obstacle, and then converts the original image frame into a corresponding bird's-eye view and obtains a fitted ground plane in combination with the camera posture pre-calibrated by the vehicle-mounted camera 1, and then calculates the three-dimensional initial information of the obstacle contained in the original image frame in combination with the bird's-eye view and the fitted ground plane. Moreover, the first center point coordinates of each obstacle detected by the vehicle-mounted radar 2 are calculated based on the obstacle data of the obstacle detected by the vehicle-mounted radar 2, so that various types of data corresponding to the same obstacle can be screened and fused to generate the three-dimensional target information of the obstacle, and finally the obstacle is rendered into the three-dimensional scene map of the motor vehicle. Since the three-dimensional target information fuses the two-dimensional initial information and the three-dimensional initial information, the position detection accuracy of the three-dimensional obstacle of the motor vehicle at a close distance can be effectively improved, and the first center point coordinates obtained by the fused obstacle data detected by the vehicle-mounted radar can further improve the position detection accuracy, reduce the false detection situation, and effectively improve the stability of the system.

[0027] In a specific implementation, each vehicle-mounted camera 1 of the motor vehicle can be a left and right two-way vehicle-mounted camera, or a front and rear vehicle-mounted camera, or of course, a four-way vehicle-mounted camera. The specific number of vehicles used can be flexibly designed according to the actual computing power cost; in addition, the vehicle-mounted camera can be a fisheye camera; in addition, if Figure 1 As shown, step S2 and step S3 can be executed simultaneously, or one step can be executed first and the other step can be executed later.

[0028] It is understandable that the camera pose pre-calibration step of each vehicle-mounted camera 1 is usually completed after the vehicle-mounted camera is installed on the motor vehicle and before it is officially delivered for use, and it does not need to be calibrated again when performing three-dimensional obstacle detection and reconstruction. When calibrating, it is necessary to know the distortion table and world prior information preset before the vehicle-mounted camera 1 leaves the factory. The camera pose (R, T) of the vehicle-mounted camera 1 can be expressed by the following formula 1: (Formula 1) Among them, R is the rotation matrix and T is the translation matrix.

[0029] In an optional embodiment of the present invention, Figure 2 As shown, step S2 specifically includes: S21: converting the original image frame into an original grayscale image; S22: extracting each image feature in the original grayscale image using a HOG algorithm model; and S23: Using the SVM algorithm model in combination with the sliding window technology to select obstacles from various image features, and determining the two-dimensional initial information corresponding to the obstacle and the type of the obstacle.

[0030] In this embodiment, the original image frame is first converted into an original grayscale image to facilitate image feature recognition and extraction. The HOG algorithm model, as a fairly mature image feature extraction method, can quickly recognize and extract image features; and the SVM algorithm model combined with the sliding window technology as a common target recognition and classification algorithm combination can quickly and accurately determine the obstacles from the image features and determine their corresponding two-dimensional initial information and type.

[0031] In an optional embodiment of the present invention, Figure 3 As shown, step S3 specifically includes: S31: converting the original image frame into a bird's-eye view in combination with the pre-calibrated camera poses of each of the vehicle-mounted cameras; S32: using a PTN (Perspective Transformer Nets) network model to process the original image frame to generate corresponding three-dimensional image information; S33: using a random sample consensus algorithm (RANSAC) to perform ground plane fitting on the three-dimensional image information to generate a fitted ground plane; and S34: Perform three-dimensional object detection based on the fitted ground plane and the bird's-eye view to obtain three-dimensional initial information of obstacles.

[0032] In this embodiment, the PTN network model, as a commonly used perspective transformation network, can learn to restore the shape of a three-dimensional object from a two-dimensional image, thereby realizing the acquisition of three-dimensional image information. Further, the random sampling consistency algorithm combined with the three-dimensional image information extracted in the previous step can quickly realize the ground plane fitting. Finally, the three-dimensional object detection can be realized through the ground plane fitting, so as to select the three-dimensional obstacle and then determine the three-dimensional initial information of the obstacle.

[0033] In an optional embodiment of the present invention, the original image frame is converted into a bird's-eye view by using a BirGAN network model combined with the pre-calibrated camera poses of each of the vehicle-mounted cameras. In this embodiment, the original image frame is converted into a bird's-eye view by using a BirGAN network model, the image conversion efficiency is high, and the subsequent image processing and recognition accuracy can be guaranteed.

[0034] In an optional embodiment of the present invention, Figure 4 As shown, step S4 specifically includes: S41: clustering the obstacle data using a K-MEANS clustering algorithm model; and S42: sequentially calculating the center points of obstacle data in the same category after clustering processing to obtain the first center point coordinates of each obstacle detected by the vehicle-mounted radar 2.

[0035] In this embodiment, the K-MEANS clustering algorithm model is used to cluster obstacle data, which can effectively overcome the defect that the radar data directly obtained by the vehicle-mounted radar 2 is relatively messy, which is beneficial to subsequent calculation and processing. Obstacle data belonging to the same category can be regarded as the same obstacle, and then the first center point coordinates of each obstacle can be calculated respectively according to each group of obstacle data belonging to the same obstacle. Specifically, the vehicle-mounted radar 2 can be an ultrasonic radar or a millimeter wave radar.

[0036] In an optional embodiment of the present invention, the two-dimensional initial information, the three-dimensional initial information and the first center point coordinates corresponding to the same obstacle are screened by the following method: Projecting the first center point coordinates, the second center point coordinates included in the two-dimensional initial information, and the three-dimensional initial information into the bird's-eye view based on the camera poses pre-calibrated based on each vehicle-mounted camera to respectively generate first projection coordinates, second projection coordinates, and a two-dimensional plane projection frame; and Determine whether the first projection coordinates and the second projection coordinates are within the two-dimensional plane projection frame. If both the first projection coordinates and the second projection coordinates are within the two-dimensional plane projection frame, determine that the corresponding first center point coordinates, the two-dimensional initial information and the two-dimensional initial information belong to the same obstacle.

[0037] In this embodiment, the data belonging to the same obstacle should be in the same approximate position. Based on this principle, by projecting the first center point coordinates, the second center point coordinates of the two-dimensional preliminary selection box contained in the two-dimensional initial information, and the three-dimensional initial information into the bird's-eye view, and judging whether the first projection coordinates and the second projection coordinates generated after projection are located in the two-dimensional plane projection box, the first center point coordinates, the two-dimensional initial information, and the three-dimensional initial information belonging to the same obstacle can be quickly and accurately screened out. Specifically, the length and width of the two-dimensional plane projection box generated by projecting the three-dimensional initial information into the bird's-eye view is the length and width of the three-dimensional preliminary selection box contained in the three-dimensional initial information.

[0038] In an optional embodiment of the present invention, the coordinates of the first center point are (x l ,y l The two-dimensional initial information includes the length w of the two-dimensional preliminary selection box containing the obstacle. 1 、Width h 1And the coordinates of the second center point (c xl , c yl The three-dimensional initial information includes the length w of the three-dimensional preliminary selection box containing the obstacle. 2 、Width h 2 , height l, coordinates of the third center point (c x2 , c y2 , c z2 ) and a rotation angle θ; the three-dimensional target information includes the length w of the three-dimensional target frame containing the obstacle x 、Width h x , height l x , target center point coordinates (c x , c y , c z ) and the rotation angle θ x ,in: w x =max(w 1 , w 2 ); h x =max(h 1 ,h 2 ); l x =l; like ,but ,otherwise ; like ,but ,otherwise ; c z = c z2 ; θ x =θ.

[0039] In this embodiment, the maximum value of the length and width in the two-dimensional preliminary selection box and the three-dimensional preliminary selection box is directly taken to ensure that the obtained three-dimensional target frame can completely contain the obstacle. As for the height, only the three-dimensional preliminary selection box has the height, and the three-dimensional target frame can directly take the height of the three-dimensional preliminary selection box and remain unchanged. For the center point coordinates, horizontal and vertical coordinates, the above rules are used for fusion processing, that is, when the difference between the xy coordinates in the center point coordinates detected by the radar and the corresponding mean values ​​of the xy coordinates of the center point coordinates in the two-dimensional preliminary selection box and the three-dimensional preliminary selection box is less than or equal to the predetermined threshold (10 cm is used as above), the mean of the three is directly used as the target center point coordinates of the three-dimensional target frame. Otherwise, if the difference exceeds the predetermined threshold, the xy coordinates in the center point coordinates detected by the radar are ignored, and the corresponding mean values ​​of the xy coordinates of the center point coordinates in the two-dimensional preliminary selection box and the three-dimensional preliminary selection box are directly taken as the target center point coordinates of the three-dimensional target frame. For the z coordinate of the center point of the three-dimensional target frame and the rotation angle, only the three-dimensional preliminary selection box has the corresponding parameter values, and the error is usually not large, so it is directly maintained unchanged.

[0040] In the specific implementation, for the rotation angle θ of the three-dimensional target frame x , can also be directly calculated by the following formula 2 to increase the rotation angle θ x Accuracy: (Formula 2).

[0041] On the other hand, Figure 5 As shown, an optional embodiment of the present invention further provides a motor vehicle three-dimensional obstacle detection and reconstruction device 3, which is connected to the vehicle-mounted camera 1 and the vehicle-mounted radar 2. The device 5 includes a processor 30, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 30. When the processor 30 executes the computer program, the motor vehicle three-dimensional obstacle detection and reconstruction method as described in any of the above embodiments is implemented.

[0042] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the motor vehicle three-dimensional obstacle detection and reconstruction device 3. For example, the computer program may be divided into Figure 6 The functional modules in the motor vehicle three-dimensional obstacle detection and reconstruction device 3, wherein the image extraction module 41, the two-dimensional information calculation module 42, the three-dimensional information calculation module 43, the radar data processing module 44, the information screening and fusion module 45 and the obstacle rendering module 46 respectively execute the above steps S1 to S6.

[0043] The motor vehicle three-dimensional obstacle detection and reconstruction device 3 can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The motor vehicle three-dimensional obstacle detection and reconstruction device 3 can include, but is not limited to, a processor 30 and a memory 32. Those skilled in the art can understand that the schematic diagram is only an example of the motor vehicle three-dimensional obstacle detection and reconstruction device 3, and does not constitute a limitation on the motor vehicle three-dimensional obstacle detection and reconstruction device 3. It can include more or less components than shown in the figure, or combine certain components, or different components. For example, the motor vehicle three-dimensional obstacle detection and reconstruction device 3 can also include input and output devices, network access devices, buses, etc.

[0044] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 30 is the control center of the motor vehicle three-dimensional obstacle detection and reconstruction device 3, and uses various interfaces and lines to connect various parts of the entire motor vehicle three-dimensional obstacle detection and reconstruction device 3.

[0045] The memory 32 can be used to store the computer program and / or module. The processor 30 realizes various functions of the motor vehicle three-dimensional obstacle detection and reconstruction device 3 by running or executing the computer program and / or module stored in the memory 32 and calling the data stored in the memory 32. The memory 32 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a graphic recognition function, a graphic overlay function, etc.), etc.; the data storage area can store data (such as graphic data, etc.) created according to the use of the control device, etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0046] If the functions described in the embodiments of the present invention are implemented in the form of software function modules or units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the embodiments of the present invention implement all or part of the processes in the above-mentioned embodiments, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 30, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0047] On the other hand, an optional embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle three-dimensional obstacle detection and reconstruction method as described in any of the above embodiments.

[0048] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0049] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, which all belong to the protection scope of the present invention.

Claims

1. A method for detecting and reconstructing three-dimensional obstacles of a motor vehicle, characterized in that: The method comprises the following steps: Extracting original image frames from video images of the surrounding environment of the motor vehicle provided by each on-board camera of the motor vehicle; Determining two-dimensional initial information of an obstacle contained in the original image frame and the type of the obstacle; Performing image processing on the original image frames in combination with the pre-calibrated camera poses of each of the vehicle-mounted cameras to obtain a fitted ground plane and a bird's-eye view corresponding to the original image frames, respectively, and calculating three-dimensional initial information of obstacles contained in the original image frames in combination with the fitted ground plane and the bird's-eye view; Calculate the first center point coordinates of each obstacle detected by the vehicle-mounted radar based on the obstacle data around the motor vehicle provided by the vehicle-mounted radar detection; Filtering and fusing the two-dimensional initial information, the three-dimensional initial information, and the first center point coordinates corresponding to the same obstacle to obtain three-dimensional target information of each obstacle; and Based on the type and three-dimensional target information of the obstacle, the obstacle is rendered into a pre-constructed three-dimensional scene map of the motor vehicle.

2. The method for detecting and reconstructing three-dimensional obstacles of a motor vehicle as claimed in claim 1, wherein: The determining of the two-dimensional initial information of the obstacle contained in the original image frame specifically includes: Converting the original image frame into an original grayscale image; Using the HOG algorithm model to extract each image feature in the original grayscale image; and The SVM algorithm model is combined with the sliding window technology to select obstacles from various image features, and the two-dimensional initial information corresponding to the obstacle and the type of the obstacle are determined.

3. The method for detecting and reconstructing three-dimensional obstacles of a motor vehicle as claimed in claim 1, wherein: The image processing of the original image frame in combination with the pre-calibrated camera poses of each of the vehicle-mounted cameras respectively obtains the fitted ground plane and the bird's-eye view corresponding to the original image frame, and calculates the three-dimensional initial information of the obstacle contained in the original image frame in combination with the fitted ground plane and the bird's-eye view specifically includes: Converting the original image frame into a bird's-eye view in combination with the pre-calibrated camera poses of each of the vehicle-mounted cameras; Using the PTN network model to process the original image frame to generate corresponding three-dimensional image information; Using a random sampling consensus algorithm to perform ground plane fitting on the three-dimensional image information to generate a fitted ground plane; and Three-dimensional object detection is performed based on the fitted ground plane in combination with the bird's-eye view to obtain three-dimensional initial information of obstacles.

4. The motor vehicle three-dimensional obstacle detection and reconstruction method according to claim 1 or 3, characterized in that: The original image frame is converted into a bird's-eye view by using a BirGAN network model combined with the pre-calibrated camera poses of each vehicle-mounted camera.

5. The motor vehicle three-dimensional obstacle detection and reconstruction method as claimed in claim 1, characterized in that: The step of calculating the first center point coordinates of each obstacle detected by the vehicle-mounted radar based on the obstacle data around the motor vehicle provided by the vehicle-mounted radar detection specifically includes: Using the K-MEANS clustering algorithm model to perform clustering processing on the obstacle data; and The center points of obstacle data in the same category after clustering are calculated in sequence to obtain the first center point coordinates of each obstacle detected by the vehicle-mounted radar.

6. The motor vehicle three-dimensional obstacle detection and reconstruction method as claimed in claim 1, characterized in that: The two-dimensional initial information, the three-dimensional initial information and the first center point coordinates corresponding to the same obstacle are screened by the following method: Projecting the first center point coordinates, the second center point coordinates included in the two-dimensional initial information, and the three-dimensional initial information into the bird's-eye view based on the camera poses pre-calibrated based on each vehicle-mounted camera to respectively generate first projection coordinates, second projection coordinates, and a two-dimensional plane projection frame; as well as Determine whether the first projection coordinates and the second projection coordinates are within the two-dimensional plane projection frame. If the first projection coordinates and the second projection coordinates are both within the two-dimensional plane projection frame, determine that the corresponding first center point coordinates, the two-dimensional initial information and the two-dimensional initial information belong to the same obstacle.

7. The motor vehicle three-dimensional obstacle detection and reconstruction method according to claim 1 or 6, characterized in that: The coordinates of the first center point are (x l ,y l The two-dimensional initial information includes the length w1, width h1 and the second center point coordinates (c xl , c yl The three-dimensional initial information includes the length w2, width h2, height l, and third center point coordinates (c x2 , c y2 , c z2 ) and a rotation angle θ; the three-dimensional target information includes the length w of the three-dimensional target frame containing the obstacle x 、Width h x , height l x , target center point coordinates (c x , c y , c z ) and the rotation angle θ x ,in: w x =max(w1,w2); h x =max(h1,h2); l x =l； like ,but ,otherwise ; like ,but ,otherwise ; c z = c z2 ; i x =θ.

8. A three-dimensional obstacle detection and reconstruction device for a motor vehicle, connected to a vehicle-mounted camera and a vehicle-mounted radar, characterized in that: The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for detecting and reconstructing three-dimensional obstacles of a motor vehicle according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the motor vehicle three-dimensional obstacle detection and reconstruction method as described in any one of claims 1 to 7.

Images