Target detection method, device and storage medium

Abstract

The present application relates to a target detection method, device and storage medium, wherein the target detection method comprises: obtaining radar detection data through a radar device, and obtaining video detection data through a video acquisition device; performing correlation matching processing on the radar detection data and the video detection data to generate an initial fusion result; determining each detected target at different heights at the same position according to the initial fusion result; calculating the equipment height of the radar device based on the radar detection data corresponding to each detected target; correcting the radar detection data according to the equipment height to obtain corrected detection data; performing correlation matching processing on the corrected detection data and the video detection data to generate a target detection result. Through the present application, the problem of low target detection accuracy is solved.

Description

Technical Field

[0001] The present application relates to the field of radar technology, and in particular to a target detection method, device and storage medium. Background Art

[0002] In the field of security, surveillance equipment based on radar and video all-in-one devices is gaining more and more attention. Although both radar equipment and video acquisition equipment can detect targets, the application scenarios of the two devices are completely different. The advantage of radar is that it can accurately obtain the spatial information and movement speed information of the target, and can detect distant targets without being affected by the environment such as lighting. The disadvantage is that it cannot obtain high-accuracy target recognition information and cannot accurately detect all targets when the vehicle is stationary. The advantage of video / image is that it can obtain high-accuracy target recognition information, and video can accurately detect all targets at medium and short distances. The disadvantage is that it is not easy to obtain accurate movement speed information and spatial position information of the target, the detection effect of distant targets is poor, and it is easily affected by poor lighting and foggy weather.

[0003] Although using radar equipment and video acquisition equipment to detect targets can complement each other's strengths and expand target detection scenarios, high-precision target recognition accuracy cannot be achieved at all times. For example, if the calibration error is too large or the fusion method is not selected properly, using both types of monitoring at the same time will reduce the accuracy of target detection data.

[0004] Currently, no effective solution has been proposed for the problem of low target detection accuracy in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a target detection method, device and storage medium to at least solve the problem of low target detection accuracy in the related art.

[0006] In a first aspect, an embodiment of the present application provides a target detection method, the method comprising:

[0007] Acquire radar detection data through radar equipment and acquire video detection data through video acquisition equipment;

[0008] Performing association matching processing on the radar detection data and the video detection data to generate an initial fusion result;

[0009] According to the initial fusion result, each measured target at different heights at the same position is determined; based on the radar detection data corresponding to each of the measured targets, the device height of the radar device is calculated;

[0010] Correcting the radar detection data according to the height of the device to obtain corrected detection data;

[0011] The corrected detection data and the video detection data are subjected to association matching processing to generate a target detection result.

[0012] In some embodiments, determining the detected targets at different heights at the same position according to the initial fusion result includes:

[0013] Determining a first fusion target and a fusion image according to the initial fusion result;

[0014] Determine pixel point position information of each of the first fusion targets in the fusion image; and determine each of the measured targets according to the pixel point position information.

[0015] In some embodiments, the measured target includes a first measured target and a second measured target; and the step of calculating the device height of the radar device based on the radar detection data corresponding to each of the measured targets includes:

[0016] Determining a first height of the first measured object and a second height of the second measured object according to the video detection data;

[0017] The device height is calculated according to the radar detection data corresponding to each of the measured targets, the first height and the second height.

[0018] In some embodiments, calculating the device height according to the radar detection data corresponding to each of the measured targets, the first height and the second height includes:

[0019] Constructing a first geometric relationship between the radar device and the first measured target according to first radar detection data corresponding to the first measured target, the horizontal distance at the same position, the device height and the first height;

[0020] constructing a second geometric relationship between the radar device and the second measured target according to the second radar detection data corresponding to the second measured target, the horizontal distance, the device height and the second height;

[0021] The device height is calculated using the first geometric relationship and the second geometric relationship.

[0022] In some embodiments, the acquiring radar detection data through a radar device and acquiring video detection data through a video acquisition device includes:

[0023] Acquire radar point cloud data collected by the radar device, and acquire image data collected by the video acquisition device;

[0024] Performing clustering processing on the radar point cloud data to obtain the radar detection data;

[0025] The target frame pixel points in the image data are calibrated to obtain the video detection data.

[0026] In some embodiments, the performing association matching processing on the radar detection data and the video detection data to obtain an initial fusion result includes:

[0027] determining a plurality of radar targets based on the radar detection data, and determining a plurality of visual targets based on the video detection data;

[0028] Establishing a first association relationship between each of the radar targets and each of the visual targets, and calculating first association degree information corresponding to each of the first association relationships based on first distance information between the radar targets and the visual targets constituting each of the first association relationships;

[0029] The initial fusion result is calculated according to the first correlation degree information.

[0030] In some embodiments, calculating the association degree information corresponding to each of the first association relationships according to the first distance information between the radar target and the visual target constituting each of the first association relationships includes:

[0031] calculating the first distance information;

[0032] When it is detected that there is a target distance greater than a preset distance threshold in the first distance information, the target distance is corrected to a preset maximum value information;

[0033] The first association degree information is calculated according to the first distance information and the maximum value information.

[0034] In some embodiments, calculating the initial fusion result according to the first association degree information includes:

[0035] A weighted calculation is performed on each of the first correlation degree information, and a target correlation relationship corresponding to the global minimum weight in each of the first correlation relationships is calculated; and the initial fusion result is calculated based on the target correlation relationship.

[0036] In some embodiments, the radar detection data is corrected according to the height of the device to obtain the corrected detection data, including:

[0037] Determine the corresponding radar target and the position information of the radar target according to the radar detection data;

[0038] The position information of the radar target is corrected according to the equipment height and the radar detection data to obtain the corrected target position information; the corrected detection data includes the corrected target position information.

[0039] In some embodiments, the performing association matching processing on the corrected detection data and the video detection data to generate a target detection result includes:

[0040] determining a plurality of correction targets based on the correction detection data, and determining a plurality of visual targets based on the video detection data;

[0041] Establishing a second association relationship between each of the correction targets and each of the visual targets, and calculating second association degree information corresponding to each of the second association relationships based on second distance information between the radar target and the visual target constituting each of the second association relationships;

[0042] The target detection result is calculated according to the second association degree information.

[0043] In some embodiments, the performing association matching processing on the corrected detection data and the video detection data to generate a target detection result includes:

[0044] Performing association matching processing on the corrected detection data and the video detection data to obtain a target fusion result;

[0045] When the target fusion result indicates that the radar-vision fusion is successful, based on the corrected detection data, attribute assignment processing is performed on the corresponding fusion target in the target fusion result to obtain a second fusion target;

[0046] When the target fusion result indicates that the radar target association fails, attribute assignment processing is performed on the fusion target corresponding to the target fusion result based on the video detection data to obtain a third fusion target;

[0047] Based on the second fusion target and the third fusion target, the target detection result is generated.

[0048] In some embodiments, the acquiring radar detection data through a radar device and acquiring video detection data through a video acquisition device includes:

[0049] In a sparse scene, the radar detection data and the video detection data are obtained; the sparse scene is a scene in which the number of scene targets is less than a preset target number threshold.

[0050] In a second aspect, an embodiment of the present application provides a target detection device, including:

[0051] An acquisition module is used to acquire radar detection data through a radar device and acquire video detection data through a video acquisition device;

[0052] A preliminary fusion module, used for performing association matching processing on the radar detection data and the video detection data to generate an initial fusion result;

[0053] A height calculation module, used to determine each measured target at different heights at the same position according to the initial fusion result; and calculate the device height of the radar device based on the radar detection data corresponding to each of the measured targets;

[0054] A correction module, used for correcting the radar detection data according to the height of the device to obtain corrected detection data;

[0055] A generation module performs association matching processing on the corrected detection data and the video detection data to generate a target detection result.

[0056] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the target detection method as described in the first aspect above is implemented.

[0057] Compared with the related art, the target detection method, device and storage medium provided in the embodiments of the present application obtain radar detection data through a radar device, and obtain video detection data through a video acquisition device; perform correlation matching processing on the radar detection data and the video detection data to generate an initial fusion result; determine each measured target at different heights in the same position based on the initial fusion result; calculate the equipment height of the radar device based on the radar detection data corresponding to each measured target; correct the radar detection data according to the equipment height to obtain corrected detection data; perform correlation matching processing on the corrected detection data and the video detection data to generate a target detection result.

[0058] Based on this, it is possible to calculate the equipment height of the radar device without human intervention, and to detect the target position by correcting the equipment height, which effectively improves the accuracy of radar visual target association. The radar target distance after the equipment height correction is used as the target position of the fusion target output, thus solving the problem that the radar device cannot measure the height information, resulting in a greater ranging error the closer the measured target is to the monitoring device. This improves the accuracy of target ranging, and also solves the problem that the current radar video target association method has poor adaptability at different installation heights.

[0059] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0061] Figure 1 is a hardware structure block diagram of a terminal of a target detection method according to an embodiment of the present application;

[0062] Figure 2 is a schematic diagram of a radar ranging according to an embodiment of the present application;

[0063] Figure 3 is a schematic diagram of a radar video surveillance scenario according to an embodiment of the present application;

[0064] Figure 4 is a flow chart of a target detection method according to an embodiment of the present application;

[0065] Figure 5 is a schematic diagram of a radar target and visual target fusion method according to an embodiment of the present application;

[0066] Figure 6 It is a schematic diagram of calculating the height of a device according to a large and small vehicle according to an embodiment of the present application;

[0067] Figure 7 is a flow chart of another target detection method according to an embodiment of the present application;

[0068] Figure 8 It is a structural block diagram of a target detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0070] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0071] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0072] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 is a hardware structure block diagram of a terminal of a target detection method according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1Different configurations shown.

[0073] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the target detection method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0075] In daily life, security has gradually received public attention. Therefore, security products are emerging in an endless stream, the application fields of security are constantly expanding, and security technology is developing rapidly. Regional monitoring technology based on radar and video is a hot topic recently. Traditional security terminal equipment is mainly visible light cameras, which are used to obtain target category information, but it is not easy to obtain target operation information.

[0076] For example, it is easy to identify the target type, but it is difficult to estimate the target's speed and spatial position. The recognition accuracy of targets at long distances, at night, and in rainy and foggy days is lower than that of targets at close range. In other words, the temporal and spatial accuracy of targets in the monitored area is inconsistent (even with significant differences). The target usually captured in the video is a rectangular area in the image.

[0077] However, radar actively emits electromagnetic waves and receives signals of the same frequency. It has a very high detection probability for moving objects or objects with a large radar reflection area. Millimeter-wave radar can work 24 hours a day and is less affected by weather, but it has a lower detection probability for stationary objects (the detection probability is not zero).

[0078] Therefore, the combination of the two can complement each other and achieve the effect of 1+1 greater than 2: it can further improve the probability of target detection and recognition, and obtain the target distance, direction and speed information, that is, obtain the target type through video, and obtain the target spatial position and speed through radar, so as to match / fuse the radar target and the video target.

[0079] It should be noted that the radar's distance measurement method is to send millimeter waves, part of which is reflected by the target, and the reflected point is then received by the device, so the target distance is calculated based on the time difference between sending and receiving. Therefore, the distance measured refers to the straight-line distance from the device to the target, while the horizontal distance from the device to the target is generally used. Please refer to Figure 2 , the distance measured by the monitoring device is c, and the horizontal distance from the target to the device is b; it can be seen that the closer to the device, the greater the deviation between the measured distance and the actual distance, while at a distance, the difference between the measured distance and the actual distance is small.

[0080] When radar and video targets are fused, accurate fusion results are the basis for subsequent event processing. Accurate fusion results rely on the accurate position mapping relationship between the two sensors in the same coordinate system. Figure 3 , the right half of the figure is the image monitoring scene, and the left half is the radar detection result and the image target mapped to the radar coordinate system. In the scene, A, B, C, and D represent four visual targets, and ①, ②, ③, and ④ represent four radar targets. The closer the radar point is to the bottom center point of the video target, the higher the fusion success rate.

[0081] exist Figure 3 In the left half, because there is no height information when the radar is measuring the distance, the closer to the monitoring device, the greater the ranging error of the radar, so the nearest radar point ① is far away from the video target A, resulting in the inability to correctly associate this target. Although there is a deviation in the distance between the slightly farther radar point ② and the video target B, they are still inside the target frame. The general fusion algorithm can correctly associate them, but there will be a large error in the ranging of this target. However, there is basically no distance deviation between the farther radar point ③ and the video target C, and the radar point ④ and the video target D. Based on the above analysis, since the current radar equipment cannot measure the height information, the closer to the monitor, the greater the ranging error, which greatly affects the accuracy of ranging and fusion.

[0082] In order to solve the above problem, this embodiment provides a target detection method. Figure 4 is a flow chart of a target detection method according to an embodiment of the present application, such as Figure 4 As shown, the process includes the following steps:

[0083] Step S410, acquiring radar detection data through a radar device, and acquiring video detection data through a video acquisition device.

[0084] The above radar detection data refers to the detection data including radar targets and radar target positions identified from the radar point cloud images of continuous frames collected by the radar equipment; the above video detection data refers to the detection data including visual targets and visual target positions identified from the image sequence of continuous frames collected by the video acquisition equipment. In order to ensure the accuracy of radar-visual fusion, the radar detection data and video detection data used for fusion should be the data collected by the radar equipment and the video acquisition equipment at the same time. Among them, the radar equipment can be a millimeter-wave radar or other equipment that realizes radio detection and ranging, and the video acquisition equipment can be a camera, video recorder or other equipment used to collect video / images.

[0085] It should be understood that the above-mentioned radar equipment and video acquisition equipment can be integrated into the same radar and video integrated equipment, or can be separately and independently set and deployed in the same installation position, which is not limited here.

[0086] Step S420: performing association matching processing on the radar detection data and the video detection data to generate an initial fusion result.

[0087] It should be noted that since radar and video sensors may collect data at different frequencies, before the radar-visual target is roughly associated, the radar detection data and video detection data in consecutive frames can also be time-stamped and synchronized to ensure that the radar detection data and video detection data used for fusion are data collected by each device at the same time. In other words, the associated targets are detected at the same time point or a similar time point.

[0088] Specifically, by clustering radar point cloud data, radar targets and their corresponding detection distances can be identified. Under normal circumstances, the target detected by radar is a clustered area, but the generative system in this paper belongs to the decision-level fusion method. For decision-level target fusion, the radar target is just a point after clustering.

[0089] Moreover, by performing target recognition on the video image sequence, the corresponding positions of the visual target and the video target can be identified. Among them, the video target is a rectangular area in the image. When locating the target, although the video target cannot directly obtain the distance information of the target, after calibrating the radar coordinate system and the visual coordinate system, the mapping position (i.e., distance) of the visual target in the radar coordinate system can be obtained. Under normal calibration conditions, the radar measurement position can only accurately describe the overall position of a target, and when there is no obstruction at close range, the visual mapping position can describe in detail the distance of a specific component of a target (such as the front, rear, and license plate).

[0090] In this step, preliminary association matching processing is performed based on the identified radar targets included in the radar detection data and the identified visual targets included in the video detection data. Among them, appropriate matching algorithms, such as nearest neighbor search, Hungarian algorithm (used to solve bipartite graph matching problems) or feature matching networks in deep learning, are used to match the targets detected by the radar with the targets detected by the video. For successfully matched targets, the information in the radar and video data is fused. This may include weighted averaging of information such as position, speed, direction, etc., or fusion based on some optimization algorithm (such as Kalman filtering, particle filtering, etc.) to obtain the above initial fusion results. The fused target will contain richer information, such as more accurate position estimation, more reliable speed measurement, etc.

[0091] Step S430, according to the initial fusion result, determine the various targets under test at different heights at the same position; and calculate the equipment height of the radar equipment based on the radar detection data corresponding to each target under test.

[0092] Among them, according to the initial fusion result, at least two targets with different heights at the same position (specifically, the position with the same horizontal distance as the monitoring equipment) are selected from the preliminary fusion targets that determine the successful fusion of radar and vision. For example, the monitoring equipment including radar equipment and video acquisition equipment is deployed at a preset height position at the traffic intersection. The monitoring equipment performs radar and vision fusion on each vehicle target passing through the traffic intersection. The fused vehicle target carries specific information such as vehicle model and vehicle height; by screening the vehicle targets at each pixel point position in each frame of the fused image, vehicle targets at the same position and with different models can be determined.

[0093] It should be understood that the detected targets such as vehicles of different heights are usually targets that are determined to pass through the same position at different times; or, if permitted by the embodiment, they may also be targets that pass through the same position at exactly the same time.

[0094] Assume that the installation position of the radar equipment is fixed and its height relative to the reference plane is unknown. By analyzing the geometric relationship between the target and the radar equipment (such as angle, distance, etc.), a mathematical model can be established to solve the height of the radar equipment. Common methods include triangulation, geometric analysis, etc. Since in the constructed mathematical models such as geometric relationships, the height of each target and its height difference can be determined through video detection data, and the only unknown parameters are the height of the radar equipment to be solved, and the horizontal distance between the target and the radar equipment (because the radar equipment measures the straight-line distance between the equipment and the target); based on this, the mathematical model constructed by two targets at different heights can be used to solve the height of the radar equipment installed.

[0095] Of course, in order to ensure the accuracy of the equipment height calculation, a larger number of measured targets at different heights can be introduced, and the algorithm can be verified in combination with measured targets at other heights; that is, the number of measured targets at different heights determined above should be at least two.

[0096] Step S440, correcting the radar detection data according to the equipment height to obtain corrected detection data.

[0097] Since the existing radar range is measured by the straight-line distance to the target, the radar equipment installed at a certain height will have a greater detection error for targets that are closer. Therefore, in this step, the radar detection data can be corrected by the equipment height of the radar equipment calculated above. Figure 2 For example, the radar detection data collected by the radar device includes the straight-line detection distance c; when the device height a of the radar device has been solved, the horizontal distance b between the ground detection target and the radar device can be solved according to the geometric relationship between the radar device and the ground detection target, that is, the corrected detection data of the radar device is calculated.

[0098] Through the above steps, the target position is corrected by the device height in view of the fact that the radar cannot perceive the height, which effectively improves the accuracy of radar visual target association.

[0099] Step S450: performing association matching processing on the corrected detection data and the video detection data to generate a target detection result.

[0100] In which, when the radar target distance corrected by the device height (that is, the corrected detection data) is used instead of the target distance directly detected by the radar device, the corrected detection data and the video detection data are again subjected to correlation matching processing. Since the more accurate corrected detection data with height correction of the radar detection data is used in the correlation matching processing at this time, this correlation matching step is a more accurate matching step with higher accuracy compared to step S420; accordingly, the target detection result generated by this fusion also eliminates the detection error caused by the device height of the radar device, ensuring the accuracy of target detection.

[0101] Specifically, the association matching process of the corrected detection data and the video detection data can be: determine the corrected position data of each radar target that has been identified based on the calculated corrected detection data; match according to the corrected position data of each radar target, and each visual target and the position data of each visual target in the above-mentioned identified video detection data; the matching process can be similar to the matching process in the above-mentioned step S420, which will not be repeated here.

[0102] It should also be noted that in the above steps S440 and S450, a more accurate radar-visual fusion may be performed on the current radar detection data and video detection data used to calculate the equipment height of the radar device. Alternatively, in another embodiment, since the equipment height of the radar device has been solved, the preliminary fusion step in step S420 may be omitted for the radar detection data and video detection data of other monitoring moments collected by the radar device and the video acquisition device, and the calculated equipment height may be directly used for correction and fine fusion processing to obtain the target detection result. It can be seen that, under the premise that the installation position of the radar device is fixed, the calculation of the equipment height of the radar device only needs to be performed once, and the calculated equipment height can be used to correct the radar detection data in subsequent multiple applications.

[0103] In the monitoring field, such as parks, roads, bridges, parks, squares, etc., the target type is obtained through video, the target spatial position and speed are obtained through radar, and the radar target and video target are correctly matched / fused. Therefore, based on this correct fusion, accurate data services can be provided for road traffic indicators (such as vehicle flow, space occupancy, time occupancy, queue length).

[0104] In the above target detection method, the targets under test at different heights at the same position are determined through the initial fusion results of radar detection data and video detection data, so that the equipment height of the radar device can be calculated without human intervention. The target position is detected by correcting the equipment height, which effectively improves the accuracy of radar visual target association. The radar target distance corrected by the equipment height is used as the target position of the fused target output, thereby solving the problem that the radar device cannot measure the height information, resulting in the larger the distance measurement error is when the target is closer to the monitoring device, improving the accuracy of target ranging, and also solving the problem that the current radar video target association method has poor adaptability at different installation heights.

[0105] In some embodiments, the obtaining of radar detection data by a radar device and the obtaining of video detection data by a video acquisition device may further include the following steps:

[0106] In a sparse scene, radar detection data and video detection data are obtained; a sparse scene is a scene where the number of scene targets is less than a preset target number threshold. Among them, when the device height of the radar device is unknown, the radar detection data and video detection data of the same target in the sparse scene can be associated to generate a fused target.

[0107] The purpose of using fusion targets in sparse scenarios is to ensure the accuracy of information and prevent errors in device height estimation caused by association errors during coarse matching, thereby ensuring the accuracy of the device height of the radar device and, in turn, the accuracy of target detection.

[0108] In some embodiments, the above-mentioned obtaining radar detection data through a radar device and obtaining video detection data through a video acquisition device include:

[0109] Obtain radar point cloud data collected by radar equipment and obtain image data collected by video acquisition equipment. Perform clustering on radar point cloud data to obtain radar detection data; perform calibration on target frame pixels in image data to obtain video detection data.

[0110] In order to better understand this application, the principle of the radar point cloud target and visual target fusion method is explained below. Figure 5 , both video intelligence and radar tracking algorithms can detect the number and position of vehicles in their respective coordinate systems (except for targets outside the monitoring range of each monitoring device and occlusion). Among them, the video tracking algorithm can give the position of the target frame in real time, and specifically establish a target sequence and a corresponding target frame sequence, for example: Vtrg1, BoxPt1; Vtrg2, BoxPt2; Vtrg3, BoxPt3; ..., Vtrgi, BoxPt i .Vtrg i Indicates the ith visual target ID, BoxPt i Indicates the position of the i-th video target frame. i is a positive integer.

[0111] About the position of the video target box BoxPt i The determination of is as follows:

[0112]

[0113] in, , , , They respectively represent the pixel coordinates of the upper left, upper right, lower left, and lower right points in the i-th video target frame.

[0114] Similarly, the radar device can detect and track the number and point cloud data of targets, and establish the following target sequence and corresponding point cloud set sequence, for example: Rtrg1, CartPtSet1; Rtrg2, CartPtSet2; Rtrg3, CartPtSet3; ..., Rtrg i ,CartPtSet i . Rtrg iIndicates radar target ID, CartPtSet i Represents a collection of radar target point cloud positions.

[0115] After processing all the measurement data with clustering and calibration algorithms, the following mapping relationship can be constructed:

[0116] Rloc i =h cluster (CartPtSet i )

[0117] Vloc i =h calib (BoxPt i )

[0118] Where Rloc i represents the position of the i-th radar target in the world coordinate system (in this paper, the radar coordinate system is used as the world coordinate system, or the data in the radar coordinate system is unified into the world coordinate system, so as to realize the coordinate system of the data collected by the radar device and the video acquisition device respectively in the future), h cluster () represents the clustering algorithm for processing radar point cloud data. i Represents the position of the i-th visual target in the world coordinate system. calib () represents the calibration function that converts the pixel points of the target box into positions in the world coordinate system.

[0119] Based on the processed radar video target information, the following mapping relationship can be constructed through matching algorithm processing:

[0120] Fp curk ,Fv curk =f(Rtrg i ,Rloc i ,……,Vtrg j ,Vloc j ,……)

[0121] Among them, Fp curk Indicates the position of the fusion target after matching, Fv curk Indicates the speed of the fused target after matching. Vtrg j Vloc represents the jth visual target; j Represents the position of the jth visual target in the world coordinate system. j is a positive integer, and the values ​​of i and j are different. The function relationship f() matches the radar target and the visual target to obtain a fused target. At this time, the fused target has information such as the position, speed, and target type of the radar and video sensors.

[0122] More specifically, the processing and clustering process of radar point cloud data includes the following steps: using density-based spatial clustering of applications with noise (DBSCAN) or other clustering methods to cluster radar point cloud data. The purpose of this step is to obtain the number and location of clustered targets based on radar point cloud data, that is, after inputting radar point cloud data, the corresponding number of radar detection targets and radar target locations, such as the number and location of vehicles, can be obtained.

[0123] The process of establishing the calibration function between video pixels and radar coordinate system includes the following steps: input the video target frame, then obtain the center point (U, V) of the target frame, and finally convert (U, V) into the corresponding output (X, Y) through the calibration algorithm, which is the actual position of the video target in the world coordinate system. The purpose of this step is to obtain the video target frame BoxPt i The actual position (X, Y) in the corresponding world coordinate system is used to unify the coordinate system of the collected data of the video acquisition device and the collected data of the radar device. Of course, there are many methods for calibrating pixel points and 2D positions, such as camera internal and external parameter calibration and four-point calibration method, which will not be repeated here.

[0124] Through the above embodiments, by acquiring radar point cloud data collected by radar equipment and performing clustering processing, and acquiring image data collected by video acquisition equipment and performing calibration processing, corresponding radar detection data and video detection data are obtained, thereby not only improving the accuracy and real-time performance of detection, but also enhancing the robustness and adaptability of the system, providing strong technical support for the development of fields such as intelligent transportation.

[0125] In some embodiments, the above-mentioned correlation matching process of the radar detection data and the video detection data to obtain the initial fusion result includes:

[0126] A plurality of radar targets are determined according to radar detection data, and a plurality of visual targets are determined according to video detection data; a first association relationship is established between each radar target and each visual target, and first association degree information corresponding to each first association relationship is calculated according to first distance information between the radar target and the visual target constituting each first association relationship; and an initial fusion result is calculated according to the first association degree information.

[0127] In this step, a coarse correlation matching is performed on the radar detection data and the video detection data of the same target, and a fused target is generated, so that the radar detection data and the video detection data of the fused target can be used in the subsequent steps to calculate the equipment height of the radar device.

[0128] The algorithm steps for coarse correlation matching of radar detection data and video detection data are as follows: ① Establish the correlation relationship between all radar targets and visual targets; ② Solve the correlation matrix through the Hungarian algorithm (Kuhn-Munkres, KM algorithm), and the matching result corresponding to the global minimum weight obtained is the best matching result.

[0129] The first association relationship between the radar targets and the visual targets is specifically used to describe whether the radar target and any visual target are the same target. For the convenience of calculation, in this embodiment, each first association relationship can be expressed in the form of a matrix, and then a first association matrix can be established as shown in Table 1:

[0130] Table 1 The first incidence matrix

[0131]

[0132] In the first correlation matrix, the rows represent radar targets and the columns represent visual targets. ij Indicates the correlation between the i-th visual target and the j-th radar target in the current frame. The correlation degree can be calculated by judging the distance between the two targets. In the previous step, the position of the video target can be mapped to the coordinates of a point in the world coordinate system (x i ,y i ), the radar target position can be expressed as (x j ,y j ), therefore, the distance model between visual target i and radar target j can be written as:

[0133]

[0134] By solving the above-mentioned first association matrix, each radar target in a frame of radar point cloud image collected by the radar device at the same time can be associated and matched with each visual target in a frame of video image collected by the video acquisition device, so as to perform information fusion on the matched radar targets and visual targets belonging to the same target to obtain an initial fusion result.

[0135] Through the above embodiment, by establishing the first association relationship between the radar target and the visual target, it can be ensured that each target can be accurately matched in the two data sources; the establishment of such an association relationship helps to reduce false positives and false negatives and improve the accuracy of detection. In addition, by calculating the first association degree information corresponding to each first association relationship, it is possible to identify and correct the erroneous or inconsistent information in a single data source, which helps to reduce the overall detection error caused by the error of a single data source.

[0136] In some embodiments, the calculation of the correlation degree information corresponding to each first correlation relationship according to the first distance information between the radar target and the visual target constituting each first correlation relationship includes:

[0137] Calculate the first distance information; if it is detected that there is a target distance greater than a preset distance threshold in the first distance information, correct the target distance to the preset maximum value information; calculate the first correlation degree information based on the first distance information and the maximum value information.

[0138] In order to prevent two targets at a long distance from each other from being misassociated, a maximum distance can be set; that is, when a distance value in the first distance information is calculated to exceed the maximum distance, the target distance Wij exceeding the maximum distance is set to a maximum value. The setting process can be shown in the following formula:

[0139] if Wij>d max then Wij=W max

[0140] In the above example, d max Represents the distance threshold, W max It should be understood that both the distance threshold and the maximum value information can be pre-set according to actual conditions.

[0141] Through the above embodiment, the maximum distance between two matching targets is limited based on the set maximum value, so that the misassociation between the radar target and the visual target can be effectively avoided, which is conducive to improving the accuracy of the initial fusion of radar and vision.

[0142] In some embodiments, the calculating of the initial fusion result according to the first correlation degree information includes:

[0143] A weighted calculation is performed on each first correlation degree information, and a target correlation relationship corresponding to the global minimum weight in each first correlation relationship is calculated; and an initial fusion result is calculated according to the target correlation relationship.

[0144] Specifically, after calculating and obtaining a plurality of first association degree information, that is, constructing and completing the first association matrix, and calculating and obtaining each element value in the first association matrix, for any matching result, there is a sum of weights. For example, when Vtrg1 is associated with Rtrg1, Vtrg2 is associated with Rtrg2, Vtrg3 is associated with Rtrg3, and Vtrg4 is associated with Rtrg4, the sum of their weights is:

[0145] W num =W 11 +W 22 +W 33 +W 44

[0146] When W num When it is the smallest, the current correlation is considered to be the best correlation result. The minimum weight matching of the correlation matrix can be solved using the KM algorithm.

[0147] Through the above embodiments, by combining weighted calculation and global minimum weight identification, the robustness of the fusion process to abnormal data and erroneous associations can be enhanced, so that even if there are partial errors or abnormal data, they can be screened and corrected through this process, thereby ensuring the reliability of the fusion result.

[0148] In some embodiments, determining the detected targets at different heights at the same position according to the initial fusion result includes:

[0149] According to the initial fusion result, the first fusion target and the fusion image are determined; the pixel position information of each first fusion target in the fusion image is determined; and according to the pixel position information, each measured target is determined.

[0150] Specifically, after solving the first association matrix through the above steps, the result of successful association between the radar target and the visual target is determined based on the initial fusion result, that is, the first fusion target is obtained. And from the first fusion target, the targets to be measured at different heights at the same position are selected respectively. Through the above embodiment, a method for determining each target to be measured based on the pixel position information in the fused image is provided, thereby ensuring the accuracy of the target to be measured screening.

[0151] In some embodiments, the above-mentioned measured target includes a first measured target and a second measured target; and calculating the device height of the radar device based on the radar detection data corresponding to each measured target includes:

[0152] According to the video detection data, a first height of the first measured target and a second height of the second measured target are determined; and according to the radar detection data, the first height and the second height corresponding to each measured target, the height of the device is calculated.

[0153] First, the first height of the first measured target and the second height of the second measured target are obtained by video detection technology; video detection technology usually uses image processing algorithms to identify and measure the height of the target. Radar detection data provides additional information about the measured target, such as distance, angle, or reflection intensity. Assume that the height of the device is the unknown number to be solved, denoted by a. The first height of the first measured target (denoted by h1) and the second height of the second measured target (denoted by h2) are known. Radar detection data provides information about the relative position between the measured target and the device. Using this information, multiple equations can be established to solve a. For example, the radar data provides the distance between the measured target and the bottom of the device (denoted by c1 and c2), and knowing the actual height of the measured target (i.e., h1 and h2), a can be calculated through geometric relationships.

[0154] In some embodiments, the calculating of the device height according to the radar detection data corresponding to each detected target, the first height, and the second height includes:

[0155] A first geometric relationship between the radar device and the first measured target is constructed based on the first radar detection data corresponding to the first measured target, the horizontal distance at the same position, the device height and the first height; a second geometric relationship between the radar device and the second measured target is constructed based on the second radar detection data corresponding to the second measured target, the horizontal distance, the device height and the second height; and the device height is calculated using the first geometric relationship and the second geometric relationship.

[0156] For example, the first target to be measured and the second target to be measured are a small car and a large car respectively; see Figure 6 , a represents the installation height of the equipment, the horizontal distance between the trolley and the small trolley is b, the actual distance of the trolley measured by the radar is c1, the actual distance of the large trolley measured by the radar is c2, the height of the trolley is h1, and the height of the large trolley is h2. Based on the above information, the following equations can be listed:

[0157]

[0158] By solving the above equation, we can calculate the unique solution for the height a of the radar equipment actually installed at the monitoring site.

[0159] Through the above embodiments, a method for estimating the height of a radar device using targets such as large and small vehicles at the same position and with different heights is provided, and the differences between targets of different heights are used to provide additional information, thereby improving the accuracy of the height estimation of the radar device.

[0160] In some embodiments, the above-mentioned associative matching processing of the corrected detection data and the video detection data to generate the target detection result includes:

[0161] Determine multiple correction targets based on the corrected detection data, and determine multiple visual targets based on the video detection data; establish a second association relationship between each correction target and each visual target, and calculate second association degree information corresponding to each second association relationship based on second distance information between the radar target and the visual target constituting each second association relationship; calculate the target detection result based on the second association degree information.

[0162] Among them, the purpose of the radar video target matching algorithm is to associate the radar detection data and video detection data of the same target and generate a fused target. The algorithm steps are: ① Establish the association matrix between all radar targets and visual targets; ② Solve the association matrix through the KM algorithm, and the matching result corresponding to the global minimum weight is the best matching result.

[0163] Specifically, the second association relationship is used to describe whether the corrected radar target (i.e., the corrected target) and any visual target are the same target. For the convenience of calculation, in this embodiment, each second association relationship can be expressed in the form of a matrix, and then a second association matrix can be established as shown in Table 2:

[0164] Table 2 The second correlation matrix

[0165]

[0166] In the first correlation matrix, the rows represent radar targets and the columns represent visual targets. ij 'Indicates the i-th visual target Vtrg in the current frame i and the jth corrected radar target Rtrg i The degree of association can be calculated by judging the distance between two targets.

[0167] In the previous step, the position of the video target can be mapped to the coordinates of a point in the world coordinate system (x i ,y i ), and the closer the detection position of the radar target is to the installation device, the greater the error between the detection position and the actual position. Therefore, the radar detection data can be corrected according to the calculated device height and the straight-line distance from the device to the target actually measured by the radar device included in the radar detection data corresponding to each radar target, to obtain the corrected detection data of the radar device. That is, in another embodiment, the radar detection data is corrected according to the device height to obtain the corrected detection data, including:

[0168] According to the radar detection data, the corresponding radar target and the position information of the radar target are determined; according to the equipment height and the radar detection data, the position information of the radar target is corrected to obtain the position information of the corrected target; the corrected detection data includes the position information of the corrected target.

[0169] Specifically, the corrected position of the radar target in the horizontal direction can be calculated by the following formula:

[0170]

[0171] Where y' represents the horizontal distance of the radar target after correction, c represents the straight-line distance from the device to the target (measured distance), and h represents the installation height of the device.

[0172] The corrected radar target position can be expressed as (x j ',y j '), therefore, the distance model between visual target i and corrected radar target j' can be written as:

[0173]

[0174] In order to prevent two targets at a long distance from each other from being misassociated, a maximum distance can be set; that is, when a distance value in the second distance information is calculated to exceed the maximum distance, the target distance Wij exceeding the maximum distance is set to a maximum value. The setting process can be shown in the following formula:

[0175] if Wij>d max then Wij=W max

[0176] In the above example, d max Represents the distance threshold, W max It should be understood that both the distance threshold and the maximum value information can be pre-set according to actual conditions.

[0177] After calculating and obtaining a plurality of second correlation degree information, that is, constructing and completing the second correlation matrix, and calculating and obtaining each element value in the second correlation matrix, there is a sum of weights for any matching result. For example, when Vtrg1 is associated with Rtrg1', Vtrg2 is associated with Rtrg2', Vtrg3 is associated with Rtrg3', and Vtrg4 is associated with Rtrg4', the sum of their weights is:

[0178] W num =W 11 '+W 22 '+W 33 '+W 44 '

[0179] When W num When ' is the smallest, the current association is considered to be the best association result. The minimum weight matching of the second association matrix can be solved using the KM algorithm.

[0180] By solving the above-mentioned second association matrix, each radar target after position correction in a frame of radar point cloud image collected by the radar device at the same time can be associated and matched with each visual target in a frame of video image collected by the video acquisition device, so as to perform information fusion on the matched radar targets and visual targets belonging to the same target and obtain the final accurate target fusion result.

[0181] In some embodiments, the above-mentioned associative matching processing of the corrected detection data and the video detection data to generate the target detection result includes:

[0182] The corrected detection data and the video detection data are associated and matched to obtain the target fusion result.

[0183] When the target fusion result indicates that the radar-visual fusion is successful, or when the target fusion result indicates that the visual target association fails, attribute assignment processing is performed on the corresponding fusion target in the target fusion result based on the corrected detection data to obtain a second fusion target.

[0184] When the target fusion result indicates that the radar target association fails, attribute assignment processing is performed on the fusion target corresponding to the target fusion result based on the video detection data to obtain a third fusion target.

[0185] Based on the second fusion target and the third fusion target, a target detection result is generated.

[0186] Specifically, after the second incidence matrix is ​​solved, the following three matching results can be obtained, and the attribute assignment methods for these three different situations are:

[0187] The first type: The radar target is successfully associated with the visual target. For the fused target generated by this type of result, its target frame uses the detection result of the visual sensor, and its position and speed use the position result of radar detection followed by device height correction.

[0188] When the radar target is successfully associated with the visual target, the fusion target uses the detection result of the visual sensor as the target frame, which usually provides higher resolution and richer texture information, helping to identify the target more accurately. At the same time, the position and speed are detected by radar and corrected by the height of the device, combining the advantages of radar in distance and speed measurement, improving the accuracy and reliability of the data.

[0189] The second type: unassociated radar targets. For the fused targets generated by this type of results, in order to ensure the integrity of the attributes of the fused targets, the target frame uses the virtual frame fitted by the radar position. The position and speed use the position result after radar detection and then device height correction.

[0190] For unassociated radar targets, the integrity of the fused target’s attributes is maintained by fitting a virtual box. Although these targets are not associated with visual targets, the position and velocity information detected by the radar and corrected by the device’s height can still provide useful data support for these targets. This method avoids the loss of important information due to lack of association.

[0191] The third type: unassociated video targets. For the fused targets generated by this type of results, the target frame uses the video detection results. To ensure the integrity of the attributes of the fused target, its position is mapped to the position in the world coordinate system using the center point of the visual frame through a calibration function. The speed of the fused target uses the speed estimated by the visual target.

[0192] For unassociated video targets, the video detection results are used as target frames, and the calibration function is used to map the center point of the visual frame to the world coordinate system, providing position information in the actual space for these targets. Although these targets are not associated with radar targets, the speed information estimated by the visual target can still provide a description of the dynamic characteristics of these targets. This strategy ensures the comprehensiveness and practicality of the fused target.

[0193] Through the above embodiments, a flexible processing method is provided for different matching situations, so that the system can cope with various complex scenarios. Whether it is a successfully associated target, an unassociated radar target or an unassociated video target, it can be reasonably processed, thereby enhancing the robustness and adaptability of the system.

[0194] The present application is described below in conjunction with specific embodiments. Figure 7 is a flow chart of another target detection method according to an embodiment of the present application. Figure 7 As shown, the process includes the following steps:

[0195] Step S701, start. The program starts. Jump to step S702.

[0196] Step S702, initialization. Setting threshold. Jump to step S703.

[0197] Step S703, camera internal and external parameters / four-point marking. The purpose of this step is to configure the parameters that the video radar calibration method relies on; the accuracy of video radar calibration affects the subsequent fusion accuracy. Jump to step S704 and step S707.

[0198] Step S704: Video target detection. Use artificial intelligence methods such as deep learning to detect the number and position of targets, and assign a unique ID to each target. Jump to step S705.

[0199] Step S705: Video target tracking. The purpose of this step is to ensure that the same target has a stable ID. Jump to step S706.

[0200] Step S706, calculate the position of the video target in the radar coordinate system, and extract the target frame position point. Based on the target frame position in the video, extract the target frame center point. And map the target frame center point to the actual position in the world coordinate system through the calibration algorithm, and jump to step S709.

[0201] Step S707, obtain radar point cloud data. The radar sensor can obtain measurements in the environment, including point cloud data from real targets and point cloud data from other false targets. Jump to step S708.

[0202] Step S708: Clustering. Using density clustering method, obtain the number of targets and the point cloud data corresponding to each target. Jump to step S709.

[0203] Step S709, time synchronization. Based on the timestamp, paired radar and video data can be obtained. Jump to step S710.

[0204] Step S710, perform rough association of radar visual targets. Jump to step S711.

[0205] Step S711, calculating the height of the equipment through the information of large and small vehicles; wherein, a reliable large and small vehicle fusion target is selected according to the radar video target matching algorithm, and the actual installation height of the radar equipment is calculated according to the above constructed equation. Jump to step S712.

[0206] Step S712, according to the above formula, correct the radar position by the device height, and jump to step S713.

[0207] Step S713, construct a radar visual target association matrix based on the radar video target precise matching algorithm, and jump to step S714.

[0208] Step S714, solve the correlation matrix by KM algorithm, and obtain three correlation results, namely, successfully correlated radar and vision, uncorrelated radar, and uncorrelated vision. Jump to step S715.

[0209] Step S715, assigning values ​​to the association result attributes. Jump to step S716.

[0210] Step S716, end.

[0211] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0212] This embodiment also provides a target detection device, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0213] Figure 8 is a structural block diagram of a target detection device according to an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0214] The acquisition module 81 is used to acquire radar detection data through the radar device and to acquire video detection data through the video acquisition device; the preliminary fusion module 82 is used to perform correlation matching processing on the radar detection data and the video detection data to generate an initial fusion result; the height calculation module 83 is used to determine the different heights of the measured targets at the same position according to the initial fusion result; based on the radar detection data corresponding to each measured target, the equipment height of the radar device is calculated; the correction module 84 is used to correct the radar detection data according to the equipment height to obtain the corrected detection data; the generation module 85 is used to perform correlation matching processing on the corrected detection data and the video detection data to generate the target detection result.

[0215] In some of the embodiments, the height calculation module 83 is also used to determine the first fusion target and the fusion image based on the initial fusion result; the height calculation module 83 is also used to determine the pixel position information of each first fusion target in the fusion image; and determine each measured target based on the pixel position information.

[0216] In some of the embodiments, the above-mentioned measured targets include a first measured target and a second measured target; the above-mentioned height calculation module 83 is also used to determine the first height of the first measured target and the second height of the second measured target based on the video detection data; the height calculation module is also used to calculate the device height based on the radar detection data, the first height and the second height corresponding to each measured target.

[0217] In some of the embodiments, the above-mentioned height calculation module 83 is also used to construct a first geometric relationship between the radar device and the first measured target based on the first radar detection data corresponding to the first measured target, the horizontal distance at the same position, the device height and the first height; the height calculation module 83 is also used to construct a second geometric relationship between the radar device and the second measured target based on the second radar detection data corresponding to the second measured target, the horizontal distance, the device height and the second height; the height calculation module 83 is also used to calculate the device height using the first geometric relationship and the second geometric relationship.

[0218] In some of the embodiments, the acquisition module 81 is also used to acquire radar point cloud data collected by a radar device and to acquire image data collected by a video acquisition device; the acquisition module 81 is also used to perform clustering processing on the radar point cloud data to obtain radar detection data; the acquisition module 81 is also used to perform calibration processing on the target frame pixel points in the image data to obtain video detection data.

[0219] In some of the embodiments, the preliminary fusion module 82 is also used to determine multiple radar targets based on radar detection data, and to determine multiple visual targets based on video detection data; the preliminary fusion module 82 is also used to establish a first association relationship between each radar target and each visual target, and to calculate the first association degree information corresponding to each first association relationship based on the first distance information between the radar target and the visual target constituting each first association relationship; the preliminary fusion module 82 is also used to calculate the initial fusion result based on the first association degree information.

[0220] In some of the embodiments, the preliminary fusion module 82 is also used to calculate the first distance information; the preliminary fusion module 82 is also used to correct the target distance to the preset maximum value information when it is detected that there is a target distance greater than a preset distance threshold in the first distance information; the preliminary fusion module 82 is also used to calculate the first correlation degree information based on the first distance information and the maximum value information.

[0221] In some embodiments, the preliminary fusion module 82 is further used to perform weighted calculation on each first correlation degree information, and calculate the target correlation relationship corresponding to the global minimum weight in each first correlation relationship; and calculate the initial fusion result according to the target correlation relationship.

[0222] In some of the embodiments, the correction module 84 is also used to determine the corresponding radar target and the position information of the radar target based on the radar detection data; the correction module 84 corrects the position information of the radar target according to the equipment height and the radar detection data to obtain the position information of the corrected target; the corrected detection data includes the position information of the corrected target.

[0223] In some of the embodiments, the above-mentioned generation module 85 is also used to determine multiple corrected targets based on the corrected detection data, and to determine multiple visual targets based on the video detection data; the generation module 85 is also used to establish a second association relationship between each corrected target and each visual target, and calculate the second association degree information corresponding to each second association relationship based on the second distance information between the radar target and the visual target constituting each second association relationship; the generation module 85 is also used to calculate the target detection result based on the second association degree information.

[0224] In some of the embodiments, the above-mentioned generation module 85 is also used to perform association matching processing on the corrected detection data and the video detection data to obtain a target fusion result; the generation module 85 is also used to perform attribute assignment processing on the corresponding fusion target in the target fusion result based on the corrected detection data to obtain a second fusion target when the target fusion result indicates that the radar-visual fusion is successful; the generation module 85 is also used to perform attribute assignment processing on the fusion target corresponding to the target fusion result based on the video detection data to obtain a third fusion target when the target fusion result indicates that the radar target association has failed; the generation module 85 is also used to generate a target detection result based on the second fusion target and the third fusion target.

[0225] In some of the embodiments, the acquisition module 81 is further used to acquire radar detection data and video detection data in a sparse scene; a sparse scene is a scene in which the number of scene targets is less than a preset target number threshold.

[0226] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0227] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0228] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0229] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0230] S1, obtain radar detection data through radar equipment, and obtain video detection data through video acquisition equipment.

[0231] S2, performing correlation matching processing on the radar detection data and the video detection data to generate an initial fusion result.

[0232] S3, according to the initial fusion result, determine the various targets under the same position at different heights; based on the radar detection data corresponding to each target under the same position, calculate the equipment height of the radar equipment.

[0233] S4, correcting the radar detection data according to the equipment height to obtain corrected detection data.

[0234] S5, performing association matching processing on the corrected detection data and the video detection data to generate a target detection result.

[0235] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0236] In addition, in combination with the target detection method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any target detection method in the above embodiment is implemented.

[0237] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0238] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0239] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0240] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire radar detection data through radar equipment and acquire video detection data through video acquisition equipment; Performing association matching processing on the radar detection data and the video detection data to generate an initial fusion result; According to the initial fusion result, each measured target at different heights at the same position is determined; based on the radar detection data corresponding to each of the measured targets, the device height of the radar device is calculated; Correcting the radar detection data according to the height of the device to obtain corrected detection data; The corrected detection data and the video detection data are subjected to association matching processing to generate a target detection result.

2. The target detection method according to claim 1, characterized in that: Determining the detected targets at different heights at the same position according to the initial fusion result includes: Determining a first fusion target and a fusion image according to the initial fusion result; Determine pixel point position information of each of the first fusion targets in the fusion image; and determine each of the measured targets according to the pixel point position information.

3. The target detection method according to claim 1, characterized in that: The measured targets include a first measured target and a second measured target; and the calculating the device height of the radar device based on the radar detection data corresponding to each of the measured targets includes: Determining a first height of the first measured object and a second height of the second measured object according to the video detection data; The device height is calculated according to the radar detection data corresponding to each of the measured targets, the first height and the second height.

4. The target detection method according to claim 3, characterized in that: The calculating the height of the device according to the radar detection data corresponding to each of the measured targets, the first height and the second height includes: Constructing a first geometric relationship between the radar device and the first measured target according to first radar detection data corresponding to the first measured target, the horizontal distance at the same position, the device height and the first height; constructing a second geometric relationship between the radar device and the second measured target according to the second radar detection data corresponding to the second measured target, the horizontal distance, the device height and the second height; The device height is calculated using the first geometric relationship and the second geometric relationship.

5. The target detection method according to claim 1, characterized in that: The obtaining of radar detection data by a radar device and obtaining of video detection data by a video acquisition device includes: Acquire radar point cloud data collected by the radar device, and acquire image data collected by the video acquisition device; Performing clustering processing on the radar point cloud data to obtain the radar detection data; The target frame pixel points in the image data are calibrated to obtain the video detection data.

6. The target detection method according to claim 1, characterized in that: The performing association matching processing on the radar detection data and the video detection data to obtain an initial fusion result includes: determining a plurality of radar targets based on the radar detection data, and determining a plurality of visual targets based on the video detection data; Establishing a first association relationship between each of the radar targets and each of the visual targets, and calculating first association degree information corresponding to each of the first association relationships based on first distance information between the radar targets and the visual targets constituting each of the first association relationships; The initial fusion result is calculated according to the first correlation degree information.

7. The target detection method according to claim 6, characterized in that: The calculating the association degree information corresponding to each of the first association relationships according to the first distance information between the radar target and the visual target constituting each of the first association relationships includes: calculating the first distance information; When it is detected that there is a target distance greater than a preset distance threshold in the first distance information, the target distance is corrected to a preset maximum value information; The first association degree information is calculated according to the first distance information and the maximum value information.

8. The target detection method according to claim 6, characterized in that: The calculating the initial fusion result according to the first correlation degree information includes: A weighted calculation is performed on each of the first correlation degree information, and a target correlation relationship corresponding to the global minimum weight in each of the first correlation relationships is calculated; and the initial fusion result is calculated based on the target correlation relationship.

9. The target detection method according to claim 1, characterized in that: The step of correcting the radar detection data according to the height of the device to obtain the corrected detection data includes: Determine the corresponding radar target and the position information of the radar target according to the radar detection data; The position information of the radar target is corrected according to the equipment height and the radar detection data to obtain the corrected target position information; the corrected detection data includes the corrected target position information.

10. The target detection method according to claim 1, characterized in that: The performing association matching processing on the corrected detection data and the video detection data to generate a target detection result includes: determining a plurality of correction targets based on the correction detection data, and determining a plurality of visual targets based on the video detection data; Establishing a second association relationship between each of the correction targets and each of the visual targets, and calculating second association degree information corresponding to each of the second association relationships based on second distance information between the radar target and the visual target constituting each of the second association relationships; The target detection result is calculated according to the second association degree information.

11. The target detection method according to claim 1, characterized in that: The performing association matching processing on the corrected detection data and the video detection data to generate a target detection result includes: Performing association matching processing on the corrected detection data and the video detection data to obtain a target fusion result; When the target fusion result indicates that the radar-vision fusion is successful, based on the corrected detection data, attribute assignment processing is performed on the corresponding fusion target in the target fusion result to obtain a second fusion target; When the target fusion result indicates that the radar target association fails, attribute assignment processing is performed on the fusion target corresponding to the target fusion result based on the video detection data to obtain a third fusion target; Based on the second fusion target and the third fusion target, the target detection result is generated.

12. The target detection method according to any one of claims 1 to 11, characterized in that: The obtaining of radar detection data by a radar device and obtaining of video detection data by a video acquisition device includes: In a sparse scene, the radar detection data and the video detection data are obtained; the sparse scene is a scene in which the number of scene targets is less than a preset target number threshold.

13. A target detection device, characterized in that: include: An acquisition module is used to acquire radar detection data through a radar device and acquire video detection data through a video acquisition device; A preliminary fusion module, used for performing association matching processing on the radar detection data and the video detection data to generate an initial fusion result; A height calculation module, used to determine each measured target at different heights at the same position according to the initial fusion result; and calculate the device height of the radar device based on the radar detection data corresponding to each of the measured targets; A correction module, used for correcting the radar detection data according to the height of the device to obtain corrected detection data; A generation module performs association matching processing on the corrected detection data and the video detection data to generate a target detection result.

14. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the target detection method according to any one of claims 1 to 12 when running.

Images