Object Detection Method, Device, Electronic Device, and Storage Medium

Abstract

An embodiment of the present invention discloses a target detection method, device, electronic device, and storage medium, including: collecting an image frame of a target object based on an in-vehicle camera of a target vehicle, and collecting a data frame of the target object based on an in-vehicle radar of the target vehicle; determining a target data frame corresponding to each image frame according to the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar; for each image frame, converting the target object in the target data frame corresponding to the image frame into the camera coordinate system to obtain the radar coordinate of the target object in the camera coordinate system; converting the radar coordinate and the camera coordinate of the target object in the image frame in the camera coordinate system into the image coordinate system through the method of converting the camera coordinate system to the image coordinate system to obtain the image coordinate of the target object in the image coordinate system; and fusing the image coordinates to obtain the position information of the target object. The technical solution of the embodiment of the present invention improves the accuracy of obstacle detection.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of assisted driving technology, and in particular, to a target detection method, device, electronic device, and storage medium. Background Art

[0002] The blind spot monitoring system can use the millimeter-wave radar behind the vehicle to detect whether there is a vehicle approaching from behind in the adjacent lane and whether there is a vehicle in the blind spot of the rearview mirror. When a vehicle approaches or there is a vehicle in the blind spot, the blind spot monitoring system will remind the driver by means of sound, light, etc.

[0003] However, due to the working principle of the millimeter-wave radar itself and the problem of low resolution, it will misidentify stationary utility poles by the roadside, fences by the roadside, stationary large trucks by the roadside, etc., and will give wrong reminders to the driver. At present, the commonly used target detection algorithms are relatively complex and can improve the target detection effect to a certain extent. However, due to the high complexity of the algorithms in terms of time and space, they exceed the computing power of the chip and cannot meet the needs of real-time driving. Summary of the Invention

[0004] Embodiments of the present invention provide a target detection method, device, electronic device, and storage medium to achieve the technical effect of improving the accuracy of obstacle detection.

[0005] In a first aspect, an embodiment of the present invention provides a target detection method, which includes:

[0006] Collecting at least one image frame of a target object based on an in-vehicle camera of a target vehicle, and collecting at least two data frames of the target object based on at least two in-vehicle radars of the target vehicle;

[0007] Determining a target data frame corresponding to each image frame according to the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar;

[0008] For each image frame, converting the target object in the target data frame corresponding to the image frame into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system;

[0009] Converting the at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system into the image coordinate system by means of converting the camera coordinate system to the image coordinate system, to obtain at least three image coordinates of the target object in the image coordinate system;

[0010] Fusing the at least three image coordinates to obtain the position information of the target object.

[0011] Second aspect, an embodiment of the present invention further provides a target detection device, which includes:

[0012] A frame acquisition module, configured to acquire at least one image frame of a target object based on an on-vehicle camera of a target vehicle, and acquire at least two data frames of the target object based on at least two on-vehicle radars of the target vehicle;

[0013] A frame synchronization module, configured to determine a target data frame corresponding to each image frame according to a camera sampling frame rate of the on-vehicle camera and a radar sampling frame rate of the on-vehicle radar;

[0014] A first coordinate conversion module, configured to, for each image frame, convert a target object in the target data frame corresponding to the image frame into a camera coordinate system, so as to obtain at least two radar coordinates of the target object in the camera coordinate system;

[0015] A second coordinate conversion module, configured to convert the at least two radar coordinates and camera coordinates of the target object in the image frame in the camera coordinate system into an image coordinate system by a method of converting the camera coordinate system to the image coordinate system, so as to obtain at least three image coordinates of the target object in the image coordinate system;

[0016] A position information determination module, configured to fuse the at least three image coordinates to obtain position information of the target object.

[0017] Third aspect, an embodiment of the present invention further provides an electronic device, and the electronic device includes:

[0018] One or more processors;

[0019] A storage device, configured to store one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the target detection method according to any one of the embodiments of the present invention.

[0021] Fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the target detection method according to any one of the embodiments of the present invention is implemented.

[0022] The technical solution of the embodiment of the present invention collects at least one image frame of a target object through an in-vehicle camera of a target vehicle, collects at least two data frames of the target object through at least two in-vehicle radars of the target vehicle, determines a target data frame corresponding to each image frame according to the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar for temporal synchronization. Furthermore, for each image frame, the target object in the target data frame corresponding to the image frame is converted into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system. The at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system are converted into the image coordinate system by the method of converting the camera coordinate system to the image coordinate system to obtain at least three image coordinates of the target object in the image coordinate system for spatial synchronization. The position information of the target object is obtained by fusing at least three image coordinates, which solves the problem of inaccurate detection of a single obstacle and the problem of excessive consumption of computing resources caused by introducing a target detection algorithm, and achieves the technical effect of improving the accuracy of obstacle detection while ensuring computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings introduced are only the drawings of a part of the embodiments to be described by the present invention, rather than all the drawings. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 It is a flowchart of a target detection method provided by Embodiment 1 of the present invention;

[0025] Figure 2 It is a schematic diagram of the installation positions of an in-vehicle camera and an in-vehicle radar provided by Embodiment 1 of the present invention;

[0026] Figure 3 It is a flowchart of a target detection method provided by Embodiment 2 of the present invention;

[0027] Figure 4 It is a flowchart of another target detection method provided by Embodiment 2 of the present invention;

[0028] Figure 5 It is a schematic diagram of the structure of a target detection device provided by Embodiment 3 of the present invention;

[0029] Figure 6 It is a schematic diagram of the structure of an electronic device provided by Embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. In addition, it should be noted that for the sake of description, only the parts related to the present invention rather than all the structures are shown in the drawings.

[0031] Embodiment 1

[0032] Figure 1 As shown in the flowchart of a target detection method provided in Embodiment 1 of the present invention, this embodiment is applicable to the situation of detecting the positions of obstacles around a target vehicle. This method can be executed by a target detection device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device. Optionally, the electronic device can be a mobile terminal, etc.

[0033] As Figure 1 described, the method of this embodiment specifically includes the following steps:

[0034] S110. Collect at least one image frame of a target object based on an on-vehicle camera of a target vehicle, and collect at least two data frames of the target object based on at least two on-vehicle radars of the target vehicle.

[0035] Among them, the target vehicle can be the vehicle currently to be detected for obstacles. The target object can be an obstacle around the target vehicle. The on-vehicle camera can be an image capturing device installed on the target vehicle. The image frame can be the frame information captured by the on-vehicle camera. The on-vehicle radar can be a millimeter-wave radar installed on the target vehicle. The data frame can be the frame information detected by the on-vehicle radar.

[0036] Specifically, obtain surrounding image frames through the on-vehicle camera of the target vehicle. The image frames include image information of the target object at the position to be detected. Through at least two on-vehicle radars of the target vehicle, based on the data frames detected by each on-vehicle radar for the surrounding environment, the data frames include data information of the target object at the position to be detected.

[0037] Optionally, the installation positions of the on-vehicle camera and the on-vehicle radar are as Figure 2 shown. The on-vehicle camera is installed inside the windshield of the target vehicle, and the on-vehicle radars are installed on both sides inside the rear bumper of the target vehicle.

[0038] S120. Determine the target data frame corresponding to each image frame according to the camera sampling frame rate of the on-vehicle camera and the radar sampling frame rate of the on-vehicle radar.

[0039] Among them, the camera sampling frame rate can be the frequency at which the on-vehicle camera obtains information, and the radar sampling frame rate can be the frequency at which the on-vehicle radar obtains information. The target data frame can be a data frame that is temporally synchronized and matched with the image frame.

[0040] Specifically, since the on-vehicle camera and the on-vehicle radar have their fixed sampling frequencies, different sampling frequencies will cause problems of data asynchronization. Therefore, the frame information collected by the on-vehicle camera and the on-vehicle radar can be synchronized in time according to the camera sampling frame rate of the on-vehicle camera and the radar sampling frame rate of the on-vehicle radar, and the target data frame corresponding to each image frame can be determined.

[0041] It should be noted that for each on-vehicle radar, a target data frame corresponding to the image frame can be determined.

[0042] S130. For each image frame, convert the target object in the target data frame corresponding to the image frame into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system.

[0043] Among them, the camera coordinate system can be the coordinate system corresponding to the on-vehicle camera. The radar coordinate system can be the coordinate system corresponding to the on-vehicle radar. The radar coordinate can be the coordinate result of the coordinate of the target object in the radar coordinate system converted into the camera coordinate system.

[0044] Specifically, in order to unify the image frame and at least two target data frames spatially, the at least two target data frames corresponding to any image frame are converted from the radar coordinate system into the camera coordinate system, and each converted result is used as a radar coordinate.

[0045] S140. Convert at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system to the image coordinate system by the method of converting the camera coordinate system to the image coordinate system, to obtain at least three image coordinates of the target object in the image coordinate system.

[0046] Among them, the camera coordinate can be the coordinate information of the target object in the image frame in the camera coordinate system. The image coordinate system can be understood as a rectangular coordinate system. The image coordinate can be the coordinate information of the target object in the image coordinate system.

[0047] Specifically, for the target object, the camera coordinates of the target object in the image frame in the camera coordinate system can be determined, and combined with the at least two radar coordinates obtained, to obtain the coordinate information of at least three target objects in the camera coordinate system. Furthermore, the coordinate information of at least three target objects in the camera coordinate system can be converted to the image coordinate system by the method of converting the camera coordinate system to the image coordinate system, to obtain at least three image coordinates.

[0048] S150. Obtain the position information of the target object by fusing at least three image coordinates.

[0049] Among them, the position information may be the position of the target object finally determined in the image coordinate system.

[0050] Specifically, after obtaining at least three image coordinates of the target object in the image coordinate system, all the image coordinates can be fused, and the fused coordinate information is used as the position information of the target object.

[0051] In the technical solution of the embodiment of the present invention, at least one image frame of the target object is collected by the on-vehicle camera of the target vehicle, at least two data frames of the target object are collected by at least two on-vehicle radars of the target vehicle, and according to the camera sampling frame rate of the on-vehicle camera and the radar sampling frame rate of the on-vehicle radar, the target data frame corresponding to each image frame is determined for temporal synchronization. Furthermore, for each image frame, the target object in the target data frame corresponding to the image frame is converted into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system. The at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system are converted into the image coordinate system by the method of converting the camera coordinate system to the image coordinate system to obtain at least three image coordinates of the target object in the image coordinate system for spatial synchronization. The position information of the target object is obtained by fusing at least three image coordinates, which solves the problem of inaccurate detection of a single obstacle and the problem of excessive consumption of computing resources caused by introducing a target detection algorithm, and achieves the technical effect of improving the accuracy of obstacle detection while ensuring the computing power.

[0052] Embodiment 2

[0053] Figure 3 FIG. is a schematic flowchart of a target detection method provided by Embodiment 2 of the present invention. On the basis of the above embodiments, for the temporal synchronization method between the image frame and the data frame, the conversion method between the radar coordinate system and the camera coordinate system, and the conversion method between the camera coordinate system and the image coordinate system, reference may be made to the technical solution of this embodiment. Among them, the explanations of the same or corresponding terms as those in the above embodiments will not be repeated here.

[0054] As Figure 3 described, the method of this embodiment specifically includes the following steps:

[0055] S210. Collect at least one image frame of the target object by the on-vehicle camera of the target vehicle, and collect at least two data frames of the target object by at least two on-vehicle radars of the target vehicle.

[0056] S220. Determine the magnitude relationship between the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar. If the camera sampling frame rate is greater than or equal to the radar sampling frame rate, then execute S230; if the camera sampling frame rate is less than the radar sampling frame rate, then execute S240.

[0057] Specifically, by determining the magnitude relationship between the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar, it can be determined whether to use the image frames collected by the in-vehicle camera or the data frames collected by the in-vehicle radar as the basis for time synchronization.

[0058] S230. For the image sampling time of each image frame, based on each radar sampling time and the image sampling time, determine the target radar sampling time that matches the image sampling time, and determine the data frame corresponding to the target radar sampling time as the target data frame corresponding to the image frame, and then execute S250.

[0059] Among them, the image sampling time can be the sampling time of each image frame, and the radar sampling time can be the sampling time of each data frame. The target radar sampling time can be a radar sampling time that is closest to the image sampling time among each radar sampling time. The target data frame can be the data frame obtained at the target radar sampling time.

[0060] Specifically, if the camera sampling frame rate is greater than or equal to the radar sampling frame rate, it indicates that the number of image frames collected by the in-vehicle camera per unit time is greater than or equal to the number of data frames collected by the in-vehicle radar. At this time, the image frames can be used as the basis to determine the target data frame corresponding to the image frames. Based on the image sampling time of each image frame, determine a radar sampling time that is closest to the image sampling time among each radar sampling time, and use the data frame corresponding to this radar sampling time as the target data frame corresponding to this image frame.

[0061] S240. For the radar sampling time of each data frame, based on the radar sampling time and each image sampling time, determine at least one target image sampling time that matches the radar sampling time, and determine the data frame as the target data frame of the image frames corresponding to each target image sampling time, and then execute S250.

[0062] Specifically, if the camera sampling frame rate is less than the radar sampling frame rate, it indicates that the number of image frames collected by the on-vehicle camera per unit time is less than the number of data frames collected by the on-vehicle radar. At this time, the data frames can be used as a reference to determine the image frames corresponding to the data frames, and the data frames can be used as the target data frames corresponding to the image frames. Based on the radar sampling time of each data frame, determine the image sampling time closest to the radar sampling time among the various image sampling times, and make the image frame corresponding to this image sampling time correspond to the data frame, and use the data frame as the target data frame corresponding to this image frame.

[0063] S250. For each image frame, determine the target distance and target angle between the on-vehicle radar and the target object according to the target object in the target data frame corresponding to the image frame.

[0064] Among them, the target distance can be the distance between the target object determined according to the target data frame and the on-vehicle radar. The target angle can be the angle between the target object determined according to the target data frame and the on-vehicle radar.

[0065] Specifically, for each image frame, the target data frame corresponding to the image frame can be determined, and the distance between the target object and the on-vehicle radar that collected the target data frame in the target data frame is determined as the target distance, and the angle between the target object and the on-vehicle radar that collected the target data frame is determined as the target angle.

[0066] S260. Determine the horizontal distance and vertical distance between the radar coordinate system and the camera coordinate system according to the radar coordinate system of the on-vehicle radar and the camera coordinate system of the on-vehicle camera.

[0067] Among them, the horizontal distance can be the distance between the horizontal plane where the radar coordinate system is located and the horizontal plane where the camera coordinate system is located, and the vertical distance can be the distance between the vertical plane where the radar coordinate system is located and the vertical plane where the camera coordinate system is located.

[0068] Specifically, since the on-vehicle radar and the on-vehicle camera are installed on the target vehicle, the positions of the radar coordinate system of the on-vehicle radar and the camera coordinate system of the on-vehicle camera in space can be determined. Furthermore, the distance between the horizontal plane where the radar coordinate system is located and the horizontal plane where the camera coordinate system is located can be used as the horizontal distance between the radar coordinate system and the camera coordinate system, and the distance between the vertical plane where the radar coordinate system is located and the vertical plane where the camera coordinate system is located can also be used as the vertical distance between the radar coordinate system and the camera coordinate system.

[0069] S270. Determine at least two radar coordinates of the target object in the camera coordinate system according to the target distance, target angle, horizontal distance, and vertical distance.

[0070] Specifically, according to the target distance, target angle, horizontal distance, and vertical distance, the coordinate information of the target object in the radar coordinate system can be converted to the camera coordinate system to obtain the converted radar coordinates. For each vehicle-mounted radar, a radar coordinate can be obtained through conversion.

[0071] Optionally, the radar coordinates of the target object in the camera coordinate system can be determined based on the following formula:

[0072]

[0073] Wherein, R represents the target distance between the vehicle-mounted radar and the target object, α represents the target angle between the vehicle-mounted radar and the target object, H represents the horizontal distance between the radar coordinate system and the camera coordinate system, L represents the vertical distance between the radar coordinate system and the camera coordinate system, X represents the abscissa of the target object in the camera coordinate system, Y represents the ordinate of the target object in the camera coordinate system, and Z represents the vertical coordinate of the target object in the camera coordinate system.

[0074] S280. Convert at least two radar coordinates and the camera coordinates of the target object in the camera coordinate system in the image frame to the image coordinate system through the method of converting from the camera coordinate system to the image coordinate system, so as to obtain at least three image coordinates of the target object in the image coordinate system.

[0075] Specifically, for the target object, the camera coordinates of the target object in the image frame in the camera coordinate system can be determined, and combined with the at least two radar coordinates obtained, the coordinate information of at least three target objects in the camera coordinate system can be obtained. Furthermore, the coordinate information of at least three target objects in the camera coordinate system can be converted to the image coordinate system through the method of converting from the camera coordinate system to the image coordinate system, so as to obtain at least three image coordinates.

[0076] Optionally, among them, the method of converting from the camera coordinate system to the image coordinate system can be:

[0077] Determine the coordinates of the target detection point in the image coordinate system according to the spatial coordinates of the target detection point in the camera coordinate system and the camera focal length of the vehicle-mounted camera.

[0078] Wherein, the target detection point is the spatial point for converting from the camera coordinate system to the image coordinate system. The spatial coordinates can be the coordinates of the target detection point in the camera coordinate system. The camera focal length can be the focal length corresponding to when the vehicle-mounted camera captures the image frame.

[0079] Optionally, determine the coordinates of the target detection point in the image coordinate system based on the following formula:

[0080]

[0081] Wherein, X represents the abscissa of the target detection point in the camera coordinate system, Y represents the ordinate of the target detection point in the camera coordinate system, Z represents the vertical coordinate of the target detection point in the camera coordinate system, T represents the camera focal length of the vehicle-mounted camera, x represents the abscissa of the target detection point in the image coordinate system, and y represents the ordinate of the target detection point in the image coordinate system.

[0082] S290. Obtain the position information of the target object by fusing at least three image coordinates.

[0083] Based on the above embodiments, Figure 4 FIG. 8 is a schematic flowchart of another target detection method provided in Embodiment 2 of the present invention.

[0084] 1. Collect millimeter-wave radar data based on millimeter-wave radar waves, and collect scene data based on a camera.

[0085] 2. Based on the millimeter-wave radar data, the height, speed, and distance of the target object can be determined, and based on the scene data, it is determined whether the current scene is a special scene.

[0086] Among them, the special scene can be a scene that requires coordinate conversion and fusion as set according to requirements. In the special scene, the camera and millimeter-wave radar waves can be used for fusion to determine the position of the target object.

[0087] 3. If the current scene belongs to the special scene, the data corresponding to the target object in the millimeter-wave radar and the data corresponding to the target object in the camera are converted into the image coordinate system, and coordinate fusion is performed to achieve target detection.

[0088] 4. If the current scene does not belong to the special scene, a conventional target detection method is used to achieve target detection.

[0089] It should be noted that the fusion of millimeter-wave radar and camera data improves the accuracy of target detection. The camera inputs the detection data of target objects in special scenes, such as stationary large vehicles, utility poles, roadside fences, etc., to the controller. At the same time, the target data collected by the millimeter-wave radar is also transmitted to the controller. After a series of processing and data comparison by the controller, the target position is finally output. If these scenes do not belong to the special scene, target detection is performed according to the conventional target detection process.

[0090] The above method can be integrated into a millimeter-wave radar and camera fusion system, which mainly includes a data acquisition module and a controller. The controller mainly includes a data storage module and a data detection module. The data acquisition module includes a camera and a millimeter-wave radar. The data acquisition module transmits the collected data to the data storage module, and the data detection module performs data comparison and processing on the data collected by the millimeter-wave radar and the camera.

[0091] Specifically, the millimeter-wave radar collects data in ordinary scenarios and transmits the speed, height, and distance of the target object to the data storage module. Since the data information of the millimeter-wave radar and the data information of the camera are not synchronized in time, time fusion is required. For example, the sampling period of the millimeter-wave radar is 50 ms, that is, the sampling frame rate is 20 frames per second, while the camera sampling frame rate is 25 frames per second. To ensure the reliability of the data, the camera sampling frame rate can be used as a reference. For each frame of image captured by the camera, the cached data corresponding to the millimeter-wave radar for this frame of image is selected, that is, a frame of radar and vision fusion data for common sampling is completed, thus ensuring the synchronization of the millimeter-wave radar data and the camera data in time. Even if the synchronization in time is achieved, spatial fusion is still required. By establishing the conversion relationship between the precise radar coordinate system, camera coordinate system, and image coordinate system, the point coordinates of the target object in the radar coordinate system can be transformed to the corresponding camera coordinate system of the camera through coordinate transformation, so as to achieve the spatial synchronization of the millimeter-wave radar and the camera. Then, the coordinates after spatial synchronization can be transformed into the image coordinate system for target detection.

[0092] The technical solution of the embodiment of the present invention is as follows: at least one image frame of a target object is collected through an in-vehicle camera of a target vehicle, at least two data frames of the target object are collected through at least two in-vehicle radars of the target vehicle, the magnitude relationship between the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar is judged. If the camera sampling frame rate is greater than or equal to the radar sampling frame rate, for the image sampling time of each image frame, according to each radar sampling time and the image sampling time, the target radar sampling time matching the image sampling time is determined, and the data frame corresponding to the target radar sampling time is determined as the target data frame corresponding to the image frame. If the camera sampling frame rate is less than the radar sampling frame rate, for the radar sampling time of each data frame, according to the radar sampling time and each image sampling time, at least one target image sampling time matching the radar sampling time is determined, and the data frame is determined as the target data frame of the image frame corresponding to each target image sampling time, so as to realize the synchronization in time of the frame data collected by the in-vehicle camera and the in-vehicle radar. Furthermore, for each image frame, according to the target object in the target data frame corresponding to the image frame, the target distance and the target angle between the in-vehicle radar and the target object are determined. According to the radar coordinate system of the in-vehicle radar and the camera coordinate system of the in-vehicle camera, the horizontal distance and the vertical distance between the radar coordinate system and the camera coordinate system are determined. According to the target distance, the target angle, the horizontal distance and the vertical distance, at least two radar coordinates of the target object in the camera coordinate system are determined, so as to realize the synchronization in space of the frame data collected by the in-vehicle camera and the in-vehicle radar. The at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system are converted into the image coordinate system by the method of converting the camera coordinate system into the image coordinate system, and at least three image coordinates of the target object in the image coordinate system are obtained. The position information of the target object is obtained by fusing the at least three image coordinates, which solves the problem of inaccurate detection of a single obstacle and the problem of excessive consumption of computing resources caused by introducing a target detection algorithm, and realizes the technical effect of improving the accuracy of obstacle detection while ensuring the computing power.

[0093] Embodiment III

[0094] Figure 5 FIG. 7 is a schematic structural diagram of a target detection device provided in Embodiment III of the present invention. The device includes: a frame acquisition module 310, a frame synchronization module 320, a first coordinate conversion module 330, a second coordinate conversion module 340, and a position information determination module 350.

[0095] Among them, the frame acquisition module 310 is used to acquire at least one image frame of the target object based on the on-vehicle camera of the target vehicle, and acquire at least two data frames of the target object based on at least two on-vehicle radars of the target vehicle; the frame synchronization module 320 is used to determine the target data frame corresponding to each image frame according to the camera sampling frame rate of the on-vehicle camera and the radar sampling frame rate of the on-vehicle radar; the first coordinate conversion module 330 is used to convert the target object in the target data frame corresponding to each image frame into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system; the second coordinate conversion module 340 is used to convert the at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system into the image coordinate system by the method of converting the camera coordinate system to the image coordinate system to obtain at least three image coordinates of the target object in the image coordinate system; the position information determination module 350 is used to fuse the at least three image coordinates to obtain the position information of the target object.

[0096] Optionally, the frame synchronization module 320 is further used to judge the magnitude relationship between the camera sampling frame rate of the on-vehicle camera and the radar sampling frame rate of the on-vehicle radar; if the camera sampling frame rate is greater than or equal to the radar sampling frame rate, then for the image sampling time of each image frame, according to each radar sampling time and the image sampling time, determine the target radar sampling time matching the image sampling time, and determine the data frame corresponding to the target radar sampling time as the target data frame corresponding to the image frame; if the camera sampling frame rate is less than the radar sampling frame rate, then for the radar sampling time of each data frame, according to the radar sampling time and each image sampling time, determine at least one target image sampling time matching the radar sampling time, and determine the data frame as the target data frame of the image frame corresponding to each target image sampling time.

[0097] Optionally, the first coordinate conversion module 330 is further used to determine the target distance and target angle between the on-vehicle radar and the target object according to the target object in the target data frame corresponding to the image frame; determine the horizontal distance and vertical distance between the radar coordinate system and the camera coordinate system according to the radar coordinate system of the on-vehicle radar and the camera coordinate system of the on-vehicle camera; determine at least two radar coordinates of the target object in the camera coordinate system according to the target distance, the target angle, the horizontal distance and the vertical distance.

[0098] Optionally, the first coordinate conversion module 330 is further used to determine the radar coordinates of the target object in the camera coordinate system based on the following formula:

[0099]

[0100] Among them, R represents the target distance between the vehicle-mounted radar and the target object, α represents the target angle between the vehicle-mounted radar and the target object, H represents the horizontal distance between the radar coordinate system and the camera coordinate system, L represents the vertical distance between the radar coordinate system and the camera coordinate system, X represents the abscissa of the target object in the camera coordinate system, Y represents the ordinate of the target object in the camera coordinate system, and Z represents the vertical coordinate of the target object in the camera coordinate system.

[0101] Optionally, the method of converting from the camera coordinate system to the image coordinate system includes:

[0102] Determine the coordinates of the target detection point in the image coordinate system according to the spatial coordinates of the target detection point in the camera coordinate system and the camera focal length of the vehicle-mounted camera; wherein, the target detection point is the spatial point for converting from the camera coordinate system to the image coordinate system.

[0103] Optionally, the method of converting from the camera coordinate system to the image coordinate system includes:

[0104] Determine the coordinates of the target detection point in the image coordinate system based on the following formula:

[0105]

[0106] Among them, X represents the abscissa of the target detection point in the camera coordinate system, Y represents the ordinate of the target detection point in the camera coordinate system, Z represents the vertical coordinate of the target detection point in the camera coordinate system, T represents the camera focal length of the vehicle-mounted camera, x represents the abscissa of the target detection point in the image coordinate system, and y represents the ordinate of the target detection point in the image coordinate system.

[0107] Optionally, the vehicle-mounted camera is installed inside the windshield of the target vehicle, and the vehicle-mounted radar is installed on both sides inside the rear bumper of the target vehicle.

[0108] The technical solution of the embodiment of the present invention is as follows: at least one image frame of a target object is collected through an in-vehicle camera of a target vehicle, at least two data frames of the target object are collected through at least two in-vehicle radars of the target vehicle, and according to the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar, a target data frame corresponding to each image frame is determined for temporal synchronization. Furthermore, for each image frame, the target object in the target data frame corresponding to the image frame is converted into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system. The at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system are converted into the image coordinate system by a method of converting the camera coordinate system to the image coordinate system to obtain at least three image coordinates of the target object in the image coordinate system for spatial synchronization. The position information of the target object is obtained by fusing at least three image coordinates, which solves the problem of inaccurate detection of a single obstacle and the problem of excessive consumption of computing resources caused by introducing a target detection algorithm, and achieves the technical effect of improving the accuracy of obstacle detection while ensuring computing power.

[0109] The target detection device provided by the embodiment of the present invention can execute the target detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0110] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiment of the present invention.

[0111] Embodiment Four

[0112] Figure 6 It is a schematic structural diagram of an electronic device provided by Embodiment Four of the present invention. Figure 6 It shows a block diagram of an exemplary electronic device 40 suitable for implementing the implementation manner of the embodiment of the present invention. Figure 6 The shown electronic device 40 is only an example and should not bring any limitation to the functions and usage scope of the embodiment of the present invention.

[0113] As Figure 6 shown, the electronic device 40 is presented in the form of a general-purpose computing device. The components of the electronic device 40 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0114] The bus 403 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor bus, or a local bus using any of the various bus architectures. By way of example, and without limitation, these architectures include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0115] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including both volatile and nonvolatile media, removable and non-removable media.

[0116] System memory 402 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. The electronic device 40 can further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, storage system 406 can be used for reading and writing to non-removable, nonvolatile magnetic media ( Figure 6 not shown and typically called a "hard disk drive"). Although Figure 6 not shown in the figures, a disk drive for reading and writing to a removable nonvolatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to the bus 403 by one or more data media interfaces. System memory 402 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of the embodiments of the present invention.

[0117] A program / utility 408 having a set (at least one) of program modules 407 can be stored, for example, in system memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a network environment. The program modules 407 typically carry out the functions and / or methods of the embodiments described herein.

[0118] The electronic device 40 can also communicate with one or more external devices 409 (such as a keyboard, a pointing device, a display 410, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 40, and / or communicate with any device that enables the electronic device 40 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 411. Moreover, the electronic device 40 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 412. As shown in the figure, the network adapter 412 communicates with other modules of the electronic device 40 through a bus 403. It should be understood that although Figure 6 not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0119] The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402, for example, implementing the target detection method provided by the embodiments of the present invention.

[0120] Embodiment Five

[0121] Embodiment Five of the present invention also provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute a target detection method when executed by a computer processor. The method includes:

[0122] Collecting at least one image frame of a target object based on an in-vehicle camera of a target vehicle, and collecting at least two data frames of the target object based on at least two in-vehicle radars of the target vehicle;

[0123] Determining a target data frame corresponding to each image frame according to the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar;

[0124] For each image frame, converting the target object in the target data frame corresponding to the image frame into a camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system;

[0125] Converting the at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system into an image coordinate system through a method of converting the camera coordinate system to the image coordinate system to obtain at least three image coordinates of the target object in the image coordinate system;

[0126] The position information of the target object is obtained by fusing the at least three image coordinates.

[0127] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0128] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0129] The program code contained on the computer-readable medium may be transmitted by any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0130] Computer program code for performing the operations of the embodiments of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected through the Internet using an Internet service provider).

[0131] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments may be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A target detection method, characterized in that, Including: Collecting at least one image frame of a target object based on an in-vehicle camera of a target vehicle, and collecting at least two data frames of the target object based on at least two in-vehicle radars of the target vehicle; Determining a target data frame corresponding to each image frame according to the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar; wherein, for each of the in-vehicle radars, a target data frame corresponding to the image frame can be determined; For each image frame, converting the target object in the target data frame corresponding to the image frame into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system; Converting the at least two radar coordinates and the camera coordinates of the target object in the image frame in the camera coordinate system into the image coordinate system by a method of converting the camera coordinate system to the image coordinate system to obtain at least three image coordinates of the target object in the image coordinate system; Fusing the at least three image coordinates to obtain the position information of the target object; Wherein, the determining a target data frame corresponding to each image frame according to the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar includes: Judging the magnitude relationship between the camera sampling frame rate of the in-vehicle camera and the radar sampling frame rate of the in-vehicle radar; If the camera sampling frame rate is greater than or equal to the radar sampling frame rate, then for the image sampling time of each image frame, determining a target radar sampling time matching the image sampling time according to each radar sampling time and the image sampling time, and determining the data frame corresponding to the target radar sampling time as the target data frame corresponding to the image frame; If the camera sampling frame rate is less than the radar sampling frame rate, then for the radar sampling time of each data frame, determining at least one target image sampling time matching the radar sampling time according to the radar sampling time and each image sampling time, and determining the data frame as the target data frame of the image frame corresponding to each target image sampling time; Wherein, the fusing the at least three image coordinates to obtain the position information of the target object includes: Performing coordinate fusion on all the image coordinates and taking the fused coordinate information as the position information of the target object.

2. The method according to claim 1, wherein The converting the target object in the target data frame corresponding to the image frame into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system includes: Determining the target distance and target angle between the in-vehicle radar and the target object according to the target object in the target data frame corresponding to the image frame; Determining the horizontal distance and vertical distance between the radar coordinate system and the camera coordinate system according to the radar coordinate system of the in-vehicle radar and the camera coordinate system of the in-vehicle camera; Determining at least two radar coordinates of the target object in the camera coordinate system according to the target distance, the target angle, the horizontal distance and the vertical distance.

3. The method according to claim 2, characterized in that, Determining at least two radar coordinates of the target object in the camera coordinate system according to the target distance, the target angle, the horizontal distance, and the vertical distance includes: Determining the radar coordinates of the target object in the camera coordinate system based on the following formula: Where, R represents the target distance between the vehicle-mounted radar and the target object, α represents the target angle between the vehicle-mounted radar and the target object, H represents the horizontal distance between the radar coordinate system and the camera coordinate system, L represents the vertical distance between the radar coordinate system and the camera coordinate system, X represents the abscissa of the target object in the camera coordinate system, Y represents the ordinate of the target object in the camera coordinate system, and Z represents the vertical coordinate of the target object in the camera coordinate system.

4. The method according to claim 1, characterized in that The method of converting from the camera coordinate system to the image coordinate system includes: Determining the coordinates of the target detection point in the image coordinate system according to the spatial coordinates of the target detection point in the camera coordinate system and the camera focal length of the vehicle-mounted camera; wherein, the target detection point is the spatial point for converting from the camera coordinate system to the image coordinate system.

5. The method according to claim 4, characterized in that, The determining the coordinates of the target detection point in the image coordinate system according to the spatial coordinates of the target detection point in the camera coordinate system and the camera focal length of the vehicle-mounted camera includes: Determining the coordinates of the target detection point in the image coordinate system based on the following formula: Where, X represents the abscissa of the target detection point in the camera coordinate system, Y represents the ordinate of the target detection point in the camera coordinate system, Z represents the vertical coordinate of the target detection point in the camera coordinate system, T represents the camera focal length of the vehicle-mounted camera, x represents the abscissa of the target detection point in the image coordinate system, and y represents the ordinate of the target detection point in the image coordinate system.

6. The method according to claim 1, wherein The vehicle-mounted camera is installed inside the windshield of the target vehicle, and the vehicle-mounted radar is installed on both sides inside the rear bumper of the target vehicle.

7. A target detection device, characterized in that, Including: A frame acquisition module, configured to acquire at least one image frame of the target object based on the vehicle-mounted camera of the target vehicle, and acquire at least two data frames of the target object based on at least two vehicle-mounted radars of the target vehicle; A frame synchronization module, configured to determine a target data frame corresponding to each image frame according to the camera sampling frame rate of the vehicle-mounted camera and the radar sampling frame rate of the vehicle-mounted radar; wherein, for each vehicle-mounted radar, a target data frame corresponding to the image frame can be determined; A first coordinate conversion module, configured to, for each image frame, convert the target object in the target data frame corresponding to the image frame into the camera coordinate system to obtain at least two radar coordinates of the target object in the camera coordinate system; A second coordinate conversion module, configured to convert the at least two radar coordinates and the camera coordinates of the target object in the camera coordinate system in the image frame into the image coordinate system by the method of converting from the camera coordinate system to the image coordinate system to obtain at least three image coordinates of the target object in the image coordinate system; A position information determination module, configured to fuse the at least three image coordinates to obtain the position information of the target object; Wherein, the frame synchronization module is specifically configured to determine the magnitude relationship between the camera sampling frame rate of the vehicle-mounted camera and the radar sampling frame rate of the vehicle-mounted radar; if the camera sampling frame rate is greater than or equal to the radar sampling frame rate, then for the image sampling time of each image frame, according to each radar sampling time and the image sampling time, determine the target radar sampling time matching the image sampling time, and determine the data frame corresponding to the target radar sampling time as the target data frame corresponding to the image frame; if the camera sampling frame rate is less than the radar sampling frame rate, then for the radar sampling time of each data frame, according to the radar sampling time and each image sampling time, determine at least one target image sampling time matching the radar sampling time, and determine the data frame as the target data frame of the image frame corresponding to each target image sampling time; Wherein, the position information determination module is specifically configured to perform coordinate fusion on all image coordinates, and use the fused coordinate information as the position information of the target object.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the target detection method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the target detection method according to any one of claims 1-6.

Images