Target fusion method and device, equipment and storage medium

Abstract

The invention discloses a target fusion method, device and equipment and a storage medium, and relates to the technical field of target detection, and the method comprises the steps: obtaining a radar detection result on a target vehicle; acquiring a camera detection result on the target vehicle; judging whether the radar detection result is matched with the camera detection result or not based on a preset matching condition; and if yes, endowing the radar detection result with the category of the camera detection result, so that the possibility of misjudgment and misrecognition due to the fact that the object is shielded or not captured by the camera can be reduced.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a target fusion method, device, equipment and storage medium. Background Art

[0002] An autonomous driving system generally includes an environmental perception module, which relies on sensors installed on the vehicle to sample raw data from the environment. The environmental perception tasks handled by the environmental perception module include 2D / 3D target detection, etc.

[0003] Most existing methods currently combine point cloud and image data captured by lidar and cameras to complete environmental perception tasks.

[0004] In vast or more complex scenes, objects may be occluded or not captured by the camera, posing severe challenges to target detection.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a target fusion method, device, equipment and storage medium, aiming to solve the technical problem that objects may be obscured or not captured by the camera, posing severe challenges to target detection.

[0007] To achieve the above objectives, the present application proposes a target fusion method, which includes:

[0008] Get radar detection results on the target vehicle;

[0009] Obtaining camera detection results on the target vehicle;

[0010] Determining whether the radar detection result matches the camera detection result based on a preset matching condition;

[0011] If so, the category of the camera detection result is assigned to the radar detection result.

[0012] In one embodiment, the step of obtaining the camera detection result on the target vehicle includes:

[0013] Acquire an input image through a camera on the target vehicle;

[0014] Classifying the input image by an object detection algorithm;

[0015] Determine whether there are objects of the preset category in the classified input image;

[0016] If yes, determine the position and / or range of the object of the preset category, and determine the bounding box and / or object center coordinates of the object;

[0017] Performing feature matching on the object by a filter-based feature matching method to obtain a detection target;

[0018] Tracking and updating the position of the detected target in subsequent input images;

[0019] The category and position of the detection target in the camera are used as the camera detection result.

[0020] In one embodiment, the step of determining whether the radar detection result matches the camera detection result based on a preset matching condition includes:

[0021] When the distance between the target vehicle and a preset traffic scene or road feature reaches a preset threshold, the camera detection result and the radar detection result within a preset range are converted to the same coordinate system, and the camera detection result and the radar detection result are recorded in a predefined data structure, wherein the data structure includes at least one of an ID, a timestamp, a true distance, a category, and an outer contour point.

[0022] In one embodiment, the step of determining whether the radar detection result matches the camera detection result based on a preset matching condition includes:

[0023] Searching the radar detection result to determine whether the radar detection result matches the camera detection result in the previous round of target fusion process;

[0024] If yes, check whether there is an ID of the camera detection result in this round. If yes, end the matching process;

[0025] If not, calculate whether the outer contour points of the radar detection result and the outer contour points of the camera detection result intersect.

[0026] and calculating the Euclidean distance between the radar detection result and the camera detection result in sequence;

[0027] If the outer contour points of the radar detection result and the outer contour points of the camera detection result intersect, and the Euclidean distance is less than a preset matching threshold, then the radar detection result matches the camera detection result.

[0028] In one embodiment, the steps before obtaining the radar detection result on the target vehicle include:

[0029] Several cameras are mounted on the target vehicle.

[0030] In one embodiment, the step of installing a plurality of cameras on the target vehicle comprises:

[0031] Installing a first camera in front of the target vehicle;

[0032] Taking the first camera as a reference, rotate the second camera counterclockwise to a preset first angle;

[0033] Taking the first camera as a reference, the third camera is installed by rotating clockwise to a preset second angle.

[0034] In one embodiment, the step of installing a plurality of cameras on the target vehicle includes:

[0035] Using a black and white checkerboard to calibrate the intrinsic parameters of the plurality of cameras;

[0036] The preset corner points of the external reference object are used as control points to construct a set of equations for the correspondence between pixels and spatial coordinates;

[0037] In combination with the intrinsic parameters of the plurality of cameras, the set of equations for the correspondence between the pixels and the space coordinates is calculated by an optimization algorithm to obtain the extrinsic parameters of the plurality of cameras.

[0038] In addition, to achieve the above purpose, the present application also proposes a target fusion device, which includes:

[0039] A data acquisition module is used to acquire radar detection results on a target vehicle; acquire camera detection results on the target vehicle;

[0040] A matching module is used to determine whether the radar detection result matches the camera detection result based on a preset matching condition; if so, assigning the category of the camera detection result to the radar detection result.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a target fusion device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the target fusion method described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the target fusion method described above are implemented.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] This application determines the consistency of radar detection results and camera detection results based on preset matching conditions, reduces the possibility of misjudgment and misidentification due to objects being blocked or not captured by the camera, and significantly improves the target detection effect. In addition, there can be multiple cameras on the target vehicle, and the camera detection results obtained can also be multiple. Based on the preset matching conditions, it is determined whether the radar detection results and multiple camera detection results match, thereby improving the perception ability of the target vehicle, reducing the possibility of misjudgment and misidentification, and significantly improving the target detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart of the first embodiment of the target fusion method of the present application is provided;

[0048] Figure 2 A schematic diagram of the camera installation position for the target fusion method of this application;

[0049] Figure 3 A schematic diagram of camera calibration for the target fusion method of this application;

[0050] Figure 4 This is a schematic diagram of the module structure of the target fusion device of the embodiment of the present application;

[0051] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the target fusion method in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: obtain the radar detection result on the target vehicle; obtain the camera detection result on the target vehicle; determine whether the radar detection result matches the camera detection result based on a preset matching condition; if so, assign the category of the camera detection result to the radar detection result.

[0056] In this embodiment, for ease of description, the target fusion system is used as the execution subject for explanation below.

[0057] Most existing methods currently combine point cloud and image data captured by lidar and cameras to complete environmental perception tasks.

[0058] In vast or more complex scenes, objects may be occluded or not captured by the camera, posing severe challenges to target detection.

[0059] To this end, the present application provides a solution, which determines the consistency of the radar detection results and the camera detection results based on the preset matching conditions, reduces the possibility of misjudgment and misidentification due to the object being blocked or not captured by the camera, and significantly improves the target detection effect. In addition, there may be multiple cameras on the target vehicle, and the camera detection results obtained may also be multiple. Based on the preset matching conditions, it is determined whether the radar detection results and the multiple camera detection results match, thereby improving the perception ability of the target vehicle, reducing the possibility of misjudgment and misidentification, and significantly improving the target detection effect.

[0060] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a target fusion device, etc., or an electronic system capable of realizing the above functions, a target fusion system, etc. The following takes the target fusion system as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the present application embodiment provides a target fusion method, referring to Figure 1 , Figure 1 A flowchart diagram of the first embodiment of the target fusion method of the present application is provided.

[0062] In this embodiment, the target fusion method includes steps S50-S60, S80-S90:

[0063] Step S50, obtaining radar detection results on the target vehicle;

[0064] Among them, radar can provide information such as the location, speed and size of the detected target.

[0065] Step S60, obtaining a camera detection result on the target vehicle;

[0066] It should be noted that several cameras may be installed on the target vehicle to obtain several camera detection results.

[0067] Furthermore, through image processing technology, the category, outline and other visual features of the detection target can be identified.

[0068] Step S80, determining whether the radar detection result matches the camera detection result based on a preset matching condition;

[0069] In order to ensure the consistency and accuracy of multi-sensor data, it is possible to determine whether the radar detection results and the camera detection results match based on preset matching conditions.

[0070] Furthermore, the matching conditions may include the similarity of the target's position, speed, appearance features, etc. For example, the overlap of spatial positions may be used to determine whether the targets are the same object, or the consistency of the targets may be confirmed by the similarity of appearance features.

[0071] Step S90: If yes, assign the category of the camera detection result to the radar detection result.

[0072] Among them, assigning the category information in the camera detection result to the radar detection result can enrich the information content of the radar detection result.

[0073] For example, if the camera detects a pedestrian, the system can use the detailed description of the camera's detection results (such as the pedestrian's movements, posture, etc.) to optimize the radar's tracking and prediction.

[0074] This embodiment provides a target fusion method, which determines the consistency of radar detection results and camera detection results based on preset matching conditions, reduces the possibility of misjudgment and misidentification due to objects being blocked or not captured by the camera, and significantly improves the target detection effect. In addition, there may be multiple cameras on the target vehicle, and the camera detection results obtained may also be multiple. Based on the preset matching conditions, it is determined whether the radar detection results and multiple camera detection results match, thereby improving the perception ability of the target vehicle, reducing the possibility of misjudgment and misidentification, and significantly improving the target detection effect.

[0075] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be repeated in the following. On this basis, step S60, obtaining the camera detection result on the target vehicle also includes steps S61 to S67:

[0076] Step S61, acquiring an input image through a camera on the target vehicle;

[0077] It should be noted that, in this embodiment, there are several cameras on the target vehicle.

[0078] Step S62, classifying the input image using a target detection algorithm;

[0079] Among them, target detection algorithms include but are not limited to YOLO, SSD, Faster R-CNN, etc.

[0080] Step S63, determining whether an object of a preset category appears in the classified input image;

[0081] Taking a single camera as an example, by inputting a frame of RGB image, the target detection algorithm classifies the current frame of image, determines whether there are objects of preset categories in the input image, and outputs a series of labels with scores indicating the possibility of objects of preset categories appearing in the input image.

[0082] Among them, the preset categories can be people, bicycles, motorcycles, cars, road signs, unknown types, etc.

[0083] Step S64, if yes, determine the position and / or range of the object of the preset category, and determine the bounding box and / or object center coordinates of the object;

[0084] The bounding box is a rectangular bounding box that can accurately define the position and size of the object in the image. The object center coordinates are the coordinates of the center point of the bounding box, which are sometimes used to indicate the position of the object.

[0085] Step S65, performing feature matching on the object by a filter-based feature matching method to obtain a detection target;

[0086] Among them, step S65, performing feature matching on the object by using a filter-based feature matching method can be subdivided into steps S651 to S653:

[0087] Step S651: extract key visual features from the image, such as color, texture, edge, etc. These features are usually obtained through various filters (such as Gaussian filters, edge detection filters, etc.).

[0088] Step S652: convert the features extracted from the image into a mathematical form that can describe its key attributes. For example, a local feature descriptor (such as SIFT, SURF, ORB, etc.) can be used to represent the position, direction, scale and appearance of the feature.

[0089] Step S653, during the target detection or tracking process, the feature descriptors extracted from the object or scene to be identified are compared and matched with the feature descriptors in the pre-stored template or database. The matching process usually involves calculating the distance or similarity measure between the features, such as Euclidean distance, Hamming distance, cosine similarity, etc.

[0090] Step S66, tracking and updating the position of the detected target in subsequent input images;

[0091] After determining the position and range of objects of a preset category, a filter-based method is used to match target features and track specific targets, and update the target position in subsequent image frames.

[0092] Step S67: taking the category and position of the detection target in the camera as the camera detection result.

[0093] This embodiment aims to solve the problem that the autonomous driving vehicle equipped with a single camera has a small perception range of the surrounding environment, resulting in the inability to fuse the obstacle targets identified by the radar and the obstacles identified by the camera. By installing multiple cameras on the vehicle, the perception range is expanded. And using target detection technology, the target in each camera is detected and classified, and then the target is fused with the radar target. The calculation method used in this embodiment is simple, fast and efficient.

[0094] In order to improve the camera perception ability of autonomous vehicles at specific locations (long-distance zebra crossings), the camera perception algorithm can usually be improved. However, these methods have the problem of complex algorithm design: projecting radar data into image space and fusing it with camera data requires complex algorithm design and computing resources, which increases the complexity of the system.

[0095] Therefore, embodiment 3 of the present application is proposed.

[0096] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S80, before judging whether the radar detection result matches the camera detection result based on the preset matching condition, also includes step S70:

[0097] Step S70, when the distance between the target vehicle and a preset traffic scene or road feature reaches a preset threshold, the camera detection result and the radar detection result within a preset range are converted to the same coordinate system, and the camera detection result and the radar detection result are recorded in a predefined data structure, wherein the data structure includes at least one of an ID, a timestamp, a true distance, a category, and an outer contour point.

[0098] Among them, the preset traffic scenes or road features include but are not limited to sidewalks, long-distance zebra crossings, one-way streets, slow lanes, waiting areas, etc.

[0099] When the distance between the target vehicle and a preset traffic scene or road feature reaches a preset threshold, for example, when the target vehicle is within 10 meters of the sidewalk, the three-camera perception fusion logic is entered, and the camera detection results and radar detection results within a range of 15 meters centered on the vehicle position are converted to the same coordinate system, such as the camera coordinate system, radar coordinate system, vehicle body coordinate system or other custom coordinate systems.

[0100] If the target vehicle has several cameras, the camera detection results and radar detection results of the several cameras are converted to the same coordinate system.

[0101] In addition, a data structure is defined to record the ID, timestamp, actual distance calculated by the algorithm, category, detected outer contour points, etc. of the camera detection results and radar detection results.

[0102] It should be noted that the outer contour points of the camera detection results are all 4 points.

[0103] This embodiment converts the detection results of the camera and the radar into the same coordinate system to ensure that the information obtained by different sensors is consistent in space, which helps to improve the overall perception capability of the target vehicle.

[0104] Based on the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S80, judging whether the radar detection result matches the camera detection result based on the preset matching condition, also includes steps S81 to S84:

[0105] Step S81, searching the radar detection result, and determining whether the radar detection result matches the camera detection result in the previous round of target fusion process;

[0106] Step S82: If yes, check whether the ID of the camera detection result exists in this round. If yes, end the matching process;

[0107] It should be noted that the system will store the matching process of each round of radar detection results and camera detection results. If the radar detection results match the camera detection results in the previous round of target fusion, no further judgment is required.

[0108] Step S83, if not, calculate whether the outer contour points of the radar detection result and the outer contour points of the camera detection result have an intersection, and calculate the Euclidean distance between the radar detection result and the camera detection result in turn;

[0109] If the radar detection results do not match the camera detection results in the previous round of target fusion, the matching process needs to be repeated in this round.

[0110] Step S84: If the outer contour points of the radar detection result and the outer contour points of the camera detection result intersect, and the Euclidean distance is less than a preset matching threshold, then the radar detection result matches the camera detection result.

[0111] For example, if the outer contour points of the radar detection result intersect with the outer contour points of the camera detection result, and the Euclidean distance is less than 1 meter, the radar detection result matches the camera detection result, and the category of the camera detection result is attached to the radar detection result to complete the perception fusion task.

[0112] This embodiment searches the radar detection results to determine whether the radar detection results match the camera detection results in the previous round of target fusion, which helps to avoid repeated detection and improve the continuity of tracking. In addition, this embodiment verifies the geometric overlap of the radar detection results and the camera detection results by calculating the spatial intersection between the outer contour points of the radar detection results and the camera detection results. At the same time, the Euclidean distance between the two is also calculated to quantify their spatial proximity. If the outer contour points of the radar detection results and the outer contour points of the camera detection results intersect, and the Euclidean distance is less than the preset matching threshold, the radar detection results match the camera detection results, thereby improving the accuracy of the matching and reducing the false detection rate and missed detection rate.

[0113] In order to improve the camera perception capability of autonomous vehicles at specific locations (long-distance zebra crossings), wide-angle cameras can also be replaced. However, replacing wide-angle cameras will significantly increase system costs and make installation and maintenance complex.

[0114] Therefore, embodiment 5 of the present application is proposed.

[0115] Based on the first embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be repeated in detail. On this basis, step S50, before obtaining the radar detection result on the target vehicle, also includes step S10:

[0116] Step S10, installing a plurality of cameras on the target vehicle.

[0117] It should be noted that in this embodiment, one camera or multiple cameras may be installed on the target vehicle.

[0118] Furthermore, to reduce costs, a monocular camera may be used instead of a wide-angle camera.

[0119] This embodiment uses a plurality of monocular cameras to expand the perception range of the perception system, thereby reducing costs.

[0120] Considering the following shortcomings of existing technologies in autonomous vehicle perception tasks:

[0121] (1) High sensor cost: Image acquisition is performed by a camera that takes photos of the vehicle's surroundings. Low-cost L1 autonomous vehicles use a single camera, while L2 autonomous vehicles use a combination of wide-angle, medium-focal-length, and telephoto cameras. The wide-angle camera has a viewing angle of about 150°, the medium-focal-length camera has a viewing angle of about 50°, and the telephoto camera has a viewing angle of only about 35°. A combination of multiple cameras can provide a wider perception range, but it will also increase the cost of vehicle production.

[0122] (2) Complex data processing: A single camera is limited by the perspective constraint. It can convert the 2D image perspective (Image View) plus ranging perception into 3D perception under the bird's-eye view. Objects under this perspective will not have the scale and occlusion problems under the image perspective. However, this method requires other types of sensor data to be synchronized to the bird's-eye view, which increases the complexity of data processing and computing resources.

[0123] To this end, Example 6 of the present application is proposed.

[0124] Based on the fifth embodiment of the present application, in the sixth embodiment of the present application, the same or similar contents as those in the fifth embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, step S10, installing a plurality of cameras on the target vehicle also includes steps S11 to S13:

[0125] Step S11, installing a first camera in front of the target vehicle;

[0126] Step S12, taking the first camera as a reference, rotating the second camera counterclockwise to a preset first angle;

[0127] Step S13, taking the first camera as a reference, rotating the third camera clockwise to a preset second angle.

[0128] In Example 6 of the present application, three monocular cameras are installed on the target vehicle.

[0129] Reference Figure 2 , Figure 2 Schematic diagram of the camera installation position for the target fusion method of this application.

[0130] exist Figure 2 In the figure, three monocular cameras are installed at the front of the vehicle.

[0131] As an implementation mode, the first camera lens faces the front of the vehicle, the second camera is rotated 30° counterclockwise based on the first camera, and the third camera is rotated 30° clockwise based on the first camera. The three cameras are placed on the same horizontal line and are 50 cm apart horizontally.

[0132] This embodiment can determine the same target in the common viewing area between cameras by determining the installation positions of the three cameras, and the error of the target position calculated by the algorithm is small. In addition, by using three low-cost monocular cameras, only the installation position needs to be considered to improve the environmental perception ability within 20 meters around the vehicle, and improve the economy and feasibility of the autonomous driving solution. In addition, each camera processes the target in its own perspective, reducing the centralized processing of multimodal data. After actual environment testing, this method improves the real-time performance and reliability of the system.

[0133] Based on the fifth embodiment of the present application, in the seventh embodiment of the present application, the same or similar contents as those in the fifth embodiment can be referred to the above description, and will not be repeated in the following. On this basis, step S10, after installing a plurality of cameras on the target vehicle, further includes steps S20 to S40:

[0134] Step S20, using a black and white checkerboard to calibrate the intrinsic parameters of the plurality of cameras;

[0135] Among them, a black and white chessboard is used to calibrate the internal parameters of the three cameras through the pinhole imaging principle to obtain the focal length, distortion parameters, etc. of the camera.

[0136] Step S30, using the preset corner points of the external reference object as control points to construct a set of equations for the correspondence between pixels and space coordinates;

[0137] Step S40, combining the intrinsic parameters of the plurality of cameras, calculating the set of equations for the correspondence between the pixels and the space coordinates through an optimization algorithm, and obtaining the extrinsic parameters of the plurality of cameras.

[0138] Reference Figure 3 , Figure 3 A schematic diagram of camera calibration for the target fusion method of this application;

[0139] like Figure 3 As shown, first calibrate the first camera, take the position of the first camera as the reference, the black dot as the small square or small barrel, measure the position of the black dot (x, y), the front of the car is always the positive value of x, and y is correctly filled in with positive and negative signs according to the left and right positions.

[0140] Then, to calibrate the second camera, you need to rotate and move the car to a certain angle and record the position of the head point ( Figure 3The current vehicle head position and the first camera vehicle head position are measured, including: x direction, y direction, and rotation angle. The above black reference object position, rotation angle, and translation distance are transformed through the model to obtain the coordinates of the reference object in the second camera's perspective, and then the second camera's external parameter calibration is performed.

[0141] Similarly, the external parameter calibration of the third camera is also the same.

[0142] This embodiment can provide accurate internal and external parameters by calibrating the internal and external parameters of the camera to support high-precision target fusion and positioning.

[0143] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the target fusion method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0144] This application also provides a target fusion device, please refer to Figure 4 , the target fusion device comprises:

[0145] The data acquisition module 10 is used to acquire the radar detection result on the target vehicle; acquire the camera detection result on the target vehicle;

[0146] The matching module 20 is used to determine whether the radar detection result matches the camera detection result based on a preset matching condition; if so, assign the category of the camera detection result to the radar detection result.

[0147] The target fusion device provided by the present application adopts the target fusion method in the above embodiment, which can solve the technical problem that the object may be blocked or not captured by the camera, which brings severe challenges to target detection. Compared with the prior art, the beneficial effects of the target fusion device provided by the present application are the same as the beneficial effects of the target fusion method provided by the above embodiment, and the other technical features in the target fusion device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0148] The present application provides a target fusion device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target fusion method in the above-mentioned embodiment one.

[0149] Reference below Figure 5, which shows a schematic diagram of the structure of a target fusion device suitable for implementing the embodiment of the present application. The target fusion device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The target fusion device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0150] like Figure 5 As shown, the target fusion device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the target fusion device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the target fusion device to communicate with other devices wirelessly or by wire to exchange data. Although the target fusion device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0151] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0152] The target fusion device provided by the present application adopts the target fusion method in the above embodiment, which can solve the technical problem that the object may be blocked or not captured by the camera, which brings severe challenges to target detection. Compared with the prior art, the beneficial effects of the target fusion device provided by the present application are the same as the beneficial effects of the target fusion method provided by the above embodiment, and the other technical features in the target fusion device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0153] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0154] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0155] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the target fusion method in the above-mentioned embodiment.

[0156] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0157] The above-mentioned computer-readable storage medium may be included in the target fusion device; or it may exist independently without being assembled into the target fusion device.

[0158] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the target fusion device, the target fusion device: obtains the radar detection result on the target vehicle; obtains the camera detection result on the target vehicle; determines whether the radar detection result matches the camera detection result based on a preset matching condition; if so, assigns the category of the camera detection result to the radar detection result.

[0159] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0162] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned target fusion method, and can solve the technical problem that objects may be blocked or not captured by the camera, which brings severe challenges to target detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the target fusion method provided in the above-mentioned embodiment, and will not be repeated here.

[0163] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A target fusion method, characterized in that: The method comprises: Get radar detection results on the target vehicle; Obtaining camera detection results on the target vehicle; Determining whether the radar detection result matches the camera detection result based on a preset matching condition; If so, the category of the camera detection result is assigned to the radar detection result.

2. The method according to claim 1, characterized in that The step of obtaining the camera detection result on the target vehicle comprises: Acquire an input image through a camera on the target vehicle; Classifying the input image by an object detection algorithm; Determine whether there are objects of the preset category in the classified input image; If yes, determine the position and / or range of the object of the preset category, and determine the bounding box and / or object center coordinates of the object; Performing feature matching on the object by a filter-based feature matching method to obtain a detection target; Tracking and updating the position of the detected target in subsequent input images; The category and position of the detection target in the camera are used as the camera detection result.

3. The method according to claim 1, characterized in that The step of determining whether the radar detection result matches the camera detection result based on a preset matching condition includes: When the distance between the target vehicle and a preset traffic scene or road feature reaches a preset threshold, the camera detection result and the radar detection result within a preset range are converted to the same coordinate system, and the camera detection result and the radar detection result are recorded in a predefined data structure, wherein the data structure includes at least one of an ID, a timestamp, a true distance, a category, and an outer contour point.

4. The method according to claim 3, characterized in that The step of judging whether the radar detection result matches the camera detection result based on a preset matching condition comprises: Searching the radar detection result to determine whether the radar detection result matches the camera detection result in the previous round of target fusion process; If yes, check whether there is an ID of the camera detection result in this round. If yes, end the matching process; If not, calculate whether the outer contour points of the radar detection result and the outer contour points of the camera detection result have an intersection, and calculate the Euclidean distance between the radar detection result and the camera detection result in turn; If the outer contour points of the radar detection result and the outer contour points of the camera detection result intersect, and the Euclidean distance is less than a preset matching threshold, then the radar detection result matches the camera detection result.

5. The method according to claim 1, characterized in that The steps before obtaining the radar detection result on the target vehicle include: Several cameras are mounted on the target vehicle.

6. The method according to claim 5, characterized in that The step of installing a plurality of cameras on the target vehicle comprises: Installing a first camera in front of the target vehicle; Taking the first camera as a reference, rotate the second camera counterclockwise to a preset first angle; Taking the first camera as a reference, the third camera is installed by rotating clockwise to a preset second angle.

7. The method according to claim 5, characterized in that The step of installing a plurality of cameras on the target vehicle includes: Using a black and white checkerboard to calibrate the intrinsic parameters of the plurality of cameras; The preset corner points of the external reference object are used as control points to construct a set of equations for the correspondence between pixels and spatial coordinates; In combination with the intrinsic parameters of the plurality of cameras, the set of equations for the correspondence between the pixels and the space coordinates is calculated by an optimization algorithm to obtain the extrinsic parameters of the plurality of cameras.

8. A target fusion device, characterized in that: The device comprises: A data acquisition module is used to acquire radar detection results on a target vehicle; acquire camera detection results on the target vehicle; A matching module is used to determine whether the radar detection result matches the camera detection result based on a preset matching condition; if so, assigning the category of the camera detection result to the radar detection result.

9. A target fusion device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the target fusion method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the target fusion method according to any one of claims 1 to 7 are implemented.

Images