Target detection method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a target detection method, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the radar detection and image recognition of a preset detection region, and obtaining a current image recognition result and a current radar detection result of each object in the preset detection region; and in response to at least two current radar detection results of the object, screening the at least two current radar detection results of the object according to historical associated information of the object to obtain a target radar detection result of the object, the historical association information comprises a corresponding relation between a historical image recognition result and a historical radar detection result of the object; and determining object information of the object based on the target radar detection result of the object and the current image recognition result. Therefore, the comprehensiveness and accuracy of target detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection processing technology, and in particular to a target detection method, an electronic device and a computer-readable storage medium. Background Art

[0002] In traditional target detection, image acquisition equipment is still the main force in the target detection area. By acquiring the detection area through image acquisition equipment, the category information of the object can be easily identified. However, with the development of intelligent transportation, target detection not only needs to detect the category information of the object, but also needs to further obtain the motion information such as the speed and position of the object. In addition, radar detection can detect the motion information of the object, but radar detection detects the motion information of the object by sending and receiving electromagnetic waves, and cannot identify the category information of the object.

[0003] Therefore, there is an urgent need for a target detection method that integrates image acquisition equipment and radar detection. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a target detection method, an electronic device and a computer-readable storage medium, which can improve the accuracy of target detection.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: a target detection method is provided, the target detection method comprising: performing radar detection and image recognition on a preset detection area respectively, and obtaining a current image recognition result and a current radar detection result of each object in the preset detection area; in response to the number of current radar detection results of an object being at least two, screening and processing at least two current radar detection results of the object according to historical association information of each object, and obtaining a target radar detection result of the object, the historical association information including a correspondence between historical image recognition results and historical radar detection results of the object; and determining the object information of the corresponding object based on the target radar detection result of each object and the current image recognition result.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above target detection method.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium including program data stored therein, wherein the program data is used to implement the above target detection method when executed by a processor.

[0008] Compared with the traditional method of detecting targets through images, which has the problem of difficulty in obtaining the motion information of the target, the above scheme provides a target detection method, which performs radar detection and image recognition on a preset detection area respectively, and obtains the current image recognition result and the current radar detection result of each object in the preset detection area; in response to the number of current radar detection results of the object being at least two, at least two current radar detection results of the object are screened and processed according to the historical association information of the object to obtain the target radar detection result of the object, and the historical association information includes the correspondence between the historical image recognition result and the historical radar detection result of the object; the object information of the object is determined based on the target radar detection result and the current image recognition result of the object. In this way, redundant current radar detection results are screened through historical association information to solve the problem of radar splitting, and the object information of the object is obtained from multiple aspects in combination with the image recognition result, thereby improving the comprehensiveness and accuracy of the object information. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 It is a schematic diagram of radar detection results and image recognition results; Figure 2 is a flowchart of an exemplary embodiment of a target detection method shown in the present application; Figure 3 yes Figure 2 FIG. 1 is a flow chart of an exemplary embodiment of step S220 in the target detection method shown; Figure 4 is a flowchart of an exemplary embodiment of a process for determining historical association information; Figure 5 is another schematic diagram of the image recognition results and radar detection results; Figure 6 is a flow chart of an exemplary embodiment of a target radar detection result determination process; Figure 7 It is a specific flow chart of an exemplary embodiment of the target detection method shown in the present application; Figure 8 is a schematic structural diagram of an exemplary embodiment of a target detection device shown in the present application; Fig. 9 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application; Fig.10It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0010] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0011] As an important technology, target detection requires comprehensive and accurate target information. Traditional target detection mainly relies on visible light cameras. The advantage of visible light cameras is that they can obtain the category information of objects, but it is not easy to obtain the motion information of objects. Radar sensors can also perform target detection. The advantage of radar sensors is that they can accurately obtain the motion information of objects and can detect objects at a long distance without being affected by the environment such as light, but they cannot obtain high-accuracy visual recognition information. Therefore, radar sensors and image sensors can be combined for target detection. Although these two methods can complement each other and improve the comprehensiveness of target detection, if there is a large calibration error or the fusion method is not selected properly, using two sensors at the same time will reduce the accuracy of target detection.

[0012] like Figure 1 As shown in the figure, there are four objects ABCD in the preset detection area. Since the image sensor detects the target related to the pixel area size of the target, the smaller the pixel area, the smaller the detection probability. Therefore, only the three nearby objects ABC may be obtained through image recognition. The radar sensor detects the target due to the radar angle, and the nearby target may not be detected. Therefore, only the three distant objects BCD may be detected by radar, and the corresponding radar detection results are ①③④. Since the radar detection result is actually generated by clustering point cloud data, there may be clustering errors or radar retention problems caused by measurement noise and tracking strategies, resulting in two radar detection results for the same object, among which radar detection result ② is the radar split point caused by clustering errors. When fusing the radar detection results and image detection results, due to the interference of radar detection result ②, the target detection result may be misdetected, for example, radar detection result ② is mistakenly regarded as object B, or it is mistakenly believed that there is an object at radar detection result ②, resulting in traffic flow calculation errors or traffic accidents.

[0013] Based on this, this application aims to solve the problem of false detection caused by radar split points and obtain the correct fusion result on this basis. Figure 1 For example, after filtering out the radar splitting points, Figure 1 Among the four objects ABCD in the figure, A has only the image recognition result, B has both the image recognition result and the radar detection result①, C has both the image recognition result and the radar detection result③, and D has only the radar detection result④. Among them, the image recognition result is to obtain a rectangular area of ​​the object in the image. After calibrating the radar coordinate system and the visual coordinate system, the mapping position (i.e., distance) of the image recognition result in the radar coordinate system can be obtained. Therefore, under normal calibration conditions, the radar detection result can accurately describe the overall position of a target, and when there is no obstruction at a close distance, the visual mapping position can describe in detail the distance of a specific component of a target (such as the front of the car, the rear of the car, and the license plate), and obtain complete and accurate object information, providing data services for road flow indicators (such as traffic flow, space occupancy, time occupancy, and queue length).

[0014] This application proposes a target detection method, an electronic device, and a computer-readable storage medium. Figure 2 , Figure 2 It is a flowchart of an exemplary embodiment of the target detection method shown in the present application.

[0015] The target detection method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The target detection method may also be executed by a target detection device. In some possible implementations, the target detection method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0016] Specifically, the target detection method of this embodiment includes the following steps: S210: Perform radar detection and image recognition on the preset detection area respectively to obtain current image recognition results and current radar detection results of each object in the preset detection area.

[0017] The preset detection area refers to an area where target detection is required. Exemplarily, the preset detection area can be a road area, and the traffic conditions in the road area are detected. Radar detection and image recognition are performed on the preset detection area respectively to obtain the target conditions in the preset detection area. In some application scenarios, radar detection and image recognition can be performed by a radar camera. The radar camera has both radar detection and video acquisition functions, and the image can be each video frame in the video acquired by the radar camera. In other application scenarios, radar detection and image recognition can also be performed on the preset detection area by separately established radar sensors and image sensors to obtain the target conditions in the preset detection area.

[0018] An object is an object with a specific shape in a preset detection area. Exemplarily, an object can be a vehicle in a preset detection area, or an animal, etc. The target detection device can recognize and process the image collected by the image sensor to obtain the object in the preset detection area; it can also analyze and process the radar point cloud data collected by the radar sensor to obtain the object in the preset detection area; in actual applications, due to the influence of the radar angle, objects in the close range may not be detected, and due to the limitations of image recognition, objects in the long-range area may not be detected. Therefore, the image collected by the image sensor and the radar point cloud data collected by the radar sensor can be combined to obtain the object in the preset detection area. It can be seen that some objects in the preset detection area may have only the current radar detection result or only the current image recognition result.

[0019] The current image recognition result may include the visual attributes of each object. Exemplarily, the current image recognition result includes the image object identifier, category, shape, color of the object, and the distance of each specific part of the object that can be described by the visual mapping position, such as the front of the car, the rear of the car, and the license plate. Among them, an object continuous detection algorithm can be used to ensure that each object has a stable image object identifier in the images collected at different time points, that is, the historical image recognition results of the same object are consistent with the image object identifier included in the current image recognition result. Among them, after performing image recognition on the preset detection area, the current image recognition result of each object is obtained, and a target sequence is established for the current image recognition result of each object. The target sequence may include an image object identifier (image object ID), the pixel area coordinate information of the object in the current image, and the object type of each object. As an example, the target sequence is as follows:

[0020] in, The image object identifier representing each object, e.g. The corresponding objects, the image object identifiers in the image recognition results at different time points are , Represents the pixel area coordinate information of each object, Indicates the object type of each object.

[0021]

[0022] in, Represents the pixel coordinates of the upper left corner of the pixel area of ​​the object in the current image. Represents the pixel coordinates of the upper-right corner of the object in the pixel area in the current image. Represents the pixel coordinates of the lower left corner of the pixel area of ​​the object in the current image. Represents the pixel coordinates of the lower-right corner of the object in the pixel area in the current image.

[0023] The current radar detection result may include the radar object identifier, motion information and location information of the object, etc. The radar object identifier is used to indicate the unique current radar detection result in all current radar detection results. If the same object corresponds to multiple current radar detection results, that is, when one object is detected as multiple objects, the radar object identifiers in the multiple current radar detection results of the same object are also different. Exemplarily, radar detection is performed on a preset detection area to obtain radar point cloud data; the radar point cloud data is clustered to obtain clustering results; the clustering results are analyzed and processed to obtain the current radar detection results of each object. Among them, a continuous object detection strategy can be adopted to ensure that each object has a stable radar object identifier at different time points. If radar splitting occurs, the split radar detection results can be removed by the historical radar object identifier of the object. Among them, after radar detection is performed on the preset detection area, each current radar detection result is obtained, and a target sequence is established for each current radar detection result. The target sequence may include a radar object identifier (radar object ID) and radar coordinate information of the radar object. As an example, the target sequence is as follows:

[0024] in, The radar object identifier representing the current radar detection result, such as The radar object identifiers of the corresponding objects in the radar detection results at different time points are If there are two radar detection results for the object at a certain time point, the real radar detection result can be found by comparing the radar object identifier. Represents the radar coordinate information of the radar object, that is, the point cloud position set of the radar object.

[0025] The target detection device can determine whether multiple current radar detection results belong to the same object based on the mapping relationship between the current image recognition results of each object and the current radar detection results. In other embodiments, in order to reduce the computational burden, according to the characteristics of radar detection, the object that produces the wrong clustering result is generally an object with a volume larger than a preset volume, such as large vehicles such as trucks. Therefore, only objects that meet the requirements can be detected to determine whether there are multiple current radar detection results. Specifically, it is determined through the image recognition result that the object type of the object belongs to the preset type, and it is determined whether there are multiple current radar detection results in the pixel area of ​​the preset type of the object in the current image. For other objects that do not belong to the preset type, the number of current radar detection results is not determined.

[0026] S220: In response to the existence of at least two current radar detection results for an object, at least two current radar detection results for the object are screened and processed according to historical association information of each object to obtain a target radar detection result of the object, wherein the historical association information includes a correspondence between historical image recognition results and historical radar detection results of the object.

[0027] The historical association information includes the correspondence between the historical image recognition results and the historical radar detection results of the object. Exemplarily, the historical association information of each object can be determined by the historical detection results of each object. When the same historical detection result appears multiple times for the same object, the historical image recognition results and historical radar detection results in the historical detection results are determined to be the historical association information of the object. As an example, if the historical image recognition result of an object is A and the historical radar detection result is 1 multiple times, it means that the correspondence between the historical image recognition result A and the historical radar detection result 1 is the historical association information of the object, which can also be called a historical association pair.

[0028] The target radar detection result represents the unique radar detection result of each object at the current time point. Exemplarily, after the current image recognition result and the current radar detection result of each object are obtained, if there are at least two current radar detection results of the object, the at least two current radar detection results of the object are screened and processed according to the historical association information of each object to obtain the target radar detection result of the object.

[0029] The target detection device detects that there are at least two current radar detection results for the same object, that is, one current image recognition result corresponds to at least two current radar detection results; in order to ensure the reliability of the fusion between the current image recognition result and the current radar detection result and the accuracy of the object information, it is necessary to remove the redundant current radar detection results and retain the correct current radar detection results. Specifically, the current radar detection results that do not conform to the historical association information of each object can be removed according to the historical association information of the object, and the current radar detection results that conform to the historical association information are determined as the target radar detection results of the object.

[0030] S230: Determine object information of the corresponding object based on the target radar detection result of the object and the current image recognition result.

[0031] Object information includes visual attributes, motion information, and coordinate information of the object. Exemplarily, the motion information and coordinate information can be determined based on the target radar detection result of the object, and the visual attributes can be determined based on the current image recognition result of the object. For an object that only has the current image recognition result, the visual attributes, motion information, and coordinate information of the object can be determined based on the current image recognition result, wherein the coordinate information can be obtained by mapping the pixel area where the object is located in the current image recognition result to the world coordinate system through a calibration function; the motion information can be estimated through the visual target to obtain the speed of the object. For an object that only has the target radar detection result, the visual attributes, motion information, and coordinate information of the object can be determined based on the target radar detection result, wherein the visual attributes can be fitted through the coordinate information in the target radar detection result to obtain a virtual box corresponding to the current image, thereby obtaining the corresponding visual attributes.

[0032] It can be seen that the target detection method of the embodiment of the present application performs radar detection and image recognition on the preset detection area respectively, and obtains the current image recognition result and current radar detection result of each object in the preset detection area; in response to the number of current radar detection results of the object being at least two, at least two current radar detection results of the object are screened and processed according to the historical association information of the object to obtain the target radar detection result of the object, and the historical association information includes the correspondence between the historical image recognition result and the historical radar detection result of the object; the object information of the object is determined based on the target radar detection result and the current image recognition result of the object. In this way, redundant current radar detection results are screened through historical association information to solve the problem of radar splitting, and the object information of the object is obtained from multiple aspects in combination with the image recognition result, thereby improving the comprehensiveness and accuracy of the object information.

[0033] Based on the above embodiments, the embodiments of the present application adopt Figure 3The flowchart details how to filter and process multiple current radar detection results of the same object to obtain the target radar detection result. Figure 3 , Figure 3 yes Figure 2 1 is a flow chart of an exemplary embodiment of step S220 in the target detection method. Specifically, in response to the number of current radar detection results of an object being at least two, step S220 selects at least two current radar detection results of the object according to the historical association information of each object to obtain the target radar detection result of the object, and the process includes the following steps: S310: Matching the current image recognition result of the object with the historical image recognition results in each historical association information to obtain a first matching result.

[0034] The first matching result refers to the matching result between the current image recognition result of the object and each historical image recognition result. Exemplarily, the current image recognition result of the object is matched with each historical image recognition result to obtain multiple first matching results. Since the object continuous detection algorithm is adopted, the image object identifier of the same object in different images (video frames) is consistent. Therefore, the matching processing method can be to match the image object identifier in the current image recognition result with the image object identifier in the historical image recognition result. If the match is successful, it means that the corresponding historical image recognition result and the current image recognition result belong to the image recognition result of the same object.

[0035] S320: In response to the first matching result indicating a successful match, the corresponding historical association information is determined as the historical association information of the object.

[0036] If the match is successful, the first matching result represents the historical association information of the successful match to determine the historical association information of the object. If the match fails, it means that there is no historical association information corresponding to the object in the historical association information. At this time, manual intervention can be performed to screen multiple current radar detection results of the object; or by calculating the distance between the radar coordinate information in multiple current radar detection results and the position information in the current image recognition result, the one with the smallest distance is selected as the target radar detection result.

[0037] S330: Matching the historical radar detection results in the historical association information of the object with the current radar detection results of the object respectively to obtain a second matching result.

[0038] The historical association information is determined based on the historical detection results of the object, indicating that the correspondence between the historical radar detection results and the historical image recognition results in the historical association information has a certain reliability. The target detection device first matches the current image recognition result with the historical image recognition results in each historical association information to find the corresponding historical association information; then, the multiple current radar detection results corresponding to the current image recognition result are screened and processed according to the historical radar detection results in the corresponding historical association information, and the current radar detection result corresponding to the historical radar detection result is found and used as the target radar detection result. Among them, since the radar detection result also adopts the object continuous detection algorithm, the radar detection results of the same object at different time points have the same radar object identifier, so the matching processing method can also be the matching of radar object identifiers, and the radar object identifiers in the multiple current radar detection results of the object are matched with the radar object identifiers in the historical radar detection results to obtain each second matching result.

[0039] S340: In response to the second matching result indicating a successful match, determining the corresponding current radar detection result as the target radar detection result of the object.

[0040] If the radar object identifiers of the multiple current radar detection results corresponding to the object are consistent with the radar object identifiers of the historical radar detection results in the historical association information, the corresponding second matching result is determined as a successful match, and the corresponding current radar detection result is determined as the target radar detection result of the object. For other current radar detection results that have not been successfully matched, it means that they are false detection results caused by radar splitting, which need to be removed and no subsequent target detection is performed. Therefore, by filtering the current radar detection results through historical association information, the stability and correctness of the radar detection results of each object can be guaranteed.

[0041] In other embodiments, the target detection device can also determine whether there is radar splitting and filter out the true current radar detection result based on the current image recognition result and the current radar detection result. As an example, image recognition can detect close objects with high accuracy, so the current image recognition result within the close range is compared with the current radar detection result to determine whether there is a clustering error in the current radar detection result and determine the true current radar detection result.

[0042] As another example, the target detection device can also perform object recognition processing on the current image collected to obtain the target pixel area of ​​each object in the current image; perform radar detection processing on the preset detection area to obtain the initial radar detection result; convert the radar coordinate information in each initial radar detection result to the image coordinate system of the current image to obtain the converted radar coordinate information; in response to the existence of at least two converted radar coordinate information located in the same target pixel area, the corresponding at least two initial radar detection results are determined as the current radar detection results of the object corresponding to the target pixel area. Thus, it is determined whether there is an object with at least two current radar detection results through the mapping relationship between the current image recognition result and the current radar detection result.

[0043] The target pixel area belongs to the current image recognition result of each object, and represents the pixel points occupied by each object. Exemplarily, the current image collected can be recognized and processed by an object recognition algorithm to obtain the target pixel area of ​​each object. The object recognition algorithm can be a deep learning model, etc. In this embodiment, since the verification of the initial radar detection result depends on the current image recognition result, it is necessary to ensure the credibility of the current image recognition result to avoid affecting the verification result. The accuracy of the current image recognition result is related to the pixel area of ​​the object. Generally, the larger the pixel area, the more object information, and the more accurate the image recognition result obtained. Therefore, the current image recognition result with a larger pixel area can be selected to verify the initial radar detection result. The target detection device can obtain the initial pixel area of ​​each object in the current image in the preset detection area; in response to the area area of ​​the initial pixel area of ​​the object in the current image being greater than the preset area threshold, the corresponding initial pixel area is used as the target pixel area. Thus, the initial pixel areas of all objects that are greater than the preset area threshold are used as target pixel areas to ensure the credibility of the target pixel areas.

[0044] The initial radar detection result may be a radar detection result determined according to the clustering result of the radar point cloud data. Exemplarily, radar detection processing is performed on a preset detection area to obtain radar point cloud data; clustering processing is performed on the radar point cloud data to obtain clustering results; and the initial radar detection result is determined according to each group of point cloud data in the clustering results. The clustering processing method includes a density-based clustering method (Density-Based Spatial Clustering of Applications with Noise, DBSCAN). Thus, the radar point cloud data is processed by clustering processing to obtain the corresponding number of objects and object positions.

[0045] Since there may be errors in the clustering of radar point cloud data, it is easy to misclassify the same object into two objects, resulting in large errors in the output initial radar detection results. Therefore, it is necessary to verify the initial radar detection results to determine whether there is a clustering error. One of the more accurate ways is to verify the initial radar detection results through the current image recognition results obtained at the same time point. Specifically, the radar coordinate information in each initial radar detection result is converted to the image coordinate system of the current image to obtain the converted radar coordinate information; then, it is determined whether there are multiple converted radar coordinate information falling into the target pixel area of ​​the same object. If so, it means that there is a high probability of clustering error, and at least two initial radar detection results falling into the same target pixel area are determined as the current radar detection results of the object corresponding to the target pixel area, thereby avoiding errors in the number of objects and generating misleading information.

[0046] Among them, the radar coordinate information in each initial radar detection result is converted to the image coordinate system of the current image, and the pixel area coordinate information in the current image recognition result is converted to the radar coordinate system through a certain calibration function. The method of establishing the calibration function between the image coordinate system and the radar coordinate system may include but is not limited to the camera internal and external parameter calibration method, the four-point calibration method, etc. The pixel area coordinate information in the current image recognition result is converted to the radar coordinate system through the calibration function. The radar coordinate system is the world coordinate system. As an example, for the pixel area coordinate information , find the center point (U, V) of the pixel area coordinate information, and convert (U, V) to the corresponding output (X, Y) through the calibration function. This is the actual position of the object in the world coordinate system. If you want to convert (X, Y) to (U, V) through the calibration function, you only need to invert the calibration matrix from the image coordinate system to the radar coordinate system obtained by the calibration function.

[0047] In other embodiments, the initial radar detection results may be subjected to a first screening process through the historical verification results of the initial radar detection results to obtain the initial radar detection results after the first screening process; and then, by determining whether at least two converted radar coordinate information fall into the same target pixel area, the objects corresponding to each initial radar detection result are corrected. The first screening process is to determine that each initial radar detection result is identified as a radar detection result generated by a clustering error in the historical verification results, and then directly remove the corresponding initial radar detection result to obtain the initial radar detection result after the first screening process.

[0048] In order to accurately remove the redundant current radar detection results corresponding to the same object and obtain the correct target radar detection results, the target detection device needs to obtain the historical association information of each object, and filter the current radar detection results by the historical association information, so as to ensure the objectivity and accuracy of the screening criteria. The historical association information is determined by the historical image recognition results and historical radar detection results of each object at different historical time points. Specifically, the target detection device obtains the number of historical time points corresponding to the target historical image recognition results and the target historical radar detection results from the historical image recognition results and historical radar detection results of the same object at each historical time point; in response to the number of historical time points being greater than a preset threshold, the corresponding relationship between the target historical image recognition results and the target historical radar detection results is determined as the historical association information of the corresponding object. Therefore, only when the number of historical time points of the target historical image recognition results and the target historical radar detection results is greater than the preset threshold, it will be confirmed as the historical association information of the object, thereby improving the reliability of the historical association information.

[0049] In this embodiment, since the object is mobile, and the image object identifier and the radar object identifier have a certain stability, obtaining the number of historical time points corresponding to the target historical image recognition results and the target historical radar detection results is actually to determine the number of target image object identifiers in the target historical image recognition results and the target radar object identifiers in the target historical radar detection results that appear at each historical time point. Exemplarily, the image object identifiers in the historical image recognition results and the radar object identifiers in the historical radar detection results of the same object at each historical time point are obtained; and the number of historical time points at which the target image object identifier and the target radar object identifier appear at the same time at each historical time point is counted. As an example, for object A, at the first historical time point, the image object identifier of its historical image recognition result is A, and the radar object identifier of its historical radar detection result is 1; at the second historical time point, the image object identifier of its historical image recognition result is A, and the radar object identifier of its historical radar detection result is 1; at the third historical time point, the image object identifier of its historical image recognition result is A, and the radar object identifier of its historical radar detection result is 2; at the fourth historical time point, the image object identifier of its historical image recognition result is A, and the radar object identifier of its historical radar detection result is 1; the image object identifier A and the radar object identifier 1 are respectively used as the target image object identifier and the target radar object identifier, and the number of historical time points at which the image object identifier A and the radar object identifier appear simultaneously at each historical time point for object A is counted to be 3; the image object identifier A and the radar object identifier 2 are respectively used as the target image object identifier and the target radar object identifier, and the number of historical time points at which the image object identifier A and the radar object identifier 2 appear simultaneously at each historical time point for object A is counted to be 1.

[0050] When the number of historical time points of the target image object identifier and the target radar object identifier is greater than a preset threshold, it indicates that there is an association between the two, and the image recognition result with the target image object identifier and the radar detection result with the target radar object identifier are the true detection results of the object, which are determined as the historical association information of the object.

[0051] by Figure 4 The flowchart shown in the figure details how to determine the historical association information of each object, as follows: In the process of target detection based on the current image recognition result and the current radar detection result of the current frame, the initial association pair, radar track information and visual track information of each object in the previous frame are obtained. The initial association pair refers to the image object identifier in the image recognition result of each object in the previous frame and the radar object identifier in the radar detection result are determined as an association pair, such as Figure 5 As shown, Figure 5is another schematic diagram of image recognition results and radar detection results, wherein image object identifier A and radar object identifier 1 are an associated pair, which can be referred to as A1, image object identifier B and radar object identifier 2 are an associated pair, which can be referred to as B2, image object identifier C and radar object identifier 3 are an associated pair, which can be referred to as C3, and image object identifier D and radar object identifier 4 are an associated pair, which can be referred to as D4. When the object moves, the object continuous detection algorithm is used so that the same image object identifier and the same radar object identifier can stably target the same object.

[0052] Update the historical association pairs, traverse the initial association pairs of the previous frame, and determine whether a new association pair appears; if the initial association pair of the previous frame exists in the historical association pairs, then the number of historical time points of the corresponding historical association pair is increased by 1; if the initial association pair of the previous frame does not exist in the historical association pairs, then the number of historical time points of the corresponding historical association pair is set to 1, for example, if D4 exists in the historical association pairs, then the number of historical time points where the association pair D4 appears is increased by 1; determine whether the number of historical time points is greater than a preset threshold, if greater than, confirm that the image object identifier and the radar object identifier in the association pair are determined as the historical association information of the corresponding object, if not greater than, continue to traverse the next initial association pair. Therefore, it should be noted that, since this embodiment determines the association pair with a number of historical time points greater than the preset threshold as the historical association information of the corresponding object, if the number of historical time points of an object without an association pair is greater than the preset threshold, the historical association information may not be set for the object temporarily, and it may be set again after it exists, or the association pair with the largest number of historical time points may be temporarily determined as the historical association information.

[0053] In other embodiments, since there is a strong correlation between the previous frame and the current frame, the correspondence between the image recognition result and the radar detection result of each object in the previous frame can be determined as the historical correlation information of the corresponding object.

[0054] In combination with the above embodiments, Figure 6 The flowchart shown in the figure details how to filter the current radar detection results of each object through historical association information, as follows: First, the historical association information, current image recognition results, and initial radar detection results of each object are obtained; Select the current image recognition result with higher credibility from the current image recognition results. Specifically, the current image recognition result of each object includes the pixel area of ​​the corresponding object in the current image, and the initial pixel area of ​​each object in the preset detection area in the current image is obtained. If the area of ​​the initial pixel area is greater than the preset area threshold, the corresponding initial pixel area is used as the target pixel area, and the current radar detection result is screened based on the target pixel area of ​​the corresponding object, which can improve the screening accuracy; Extracting a target pixel region of a target object from the target pixel region. Exemplarily, if the object type of an object is a large vehicle, and the area of ​​the pixel region corresponding to the object is greater than a preset area threshold, it is taken as the target pixel region of the target object; Traverse all initial radar detection results, determine whether the number of times in the historical frame that the initial radar detection result is confirmed as a radar detection result caused by clustering error is greater than the number threshold, and if so, remove the initial radar detection result, thereby preventing the radar detection result caused by clustering error from being effectively identified due to visual occlusion or other visual detection abnormalities, thereby improving the recognition accuracy; if not, determine whether the initial radar detection result is in the target pixel area of ​​the target object, specifically, convert the radar coordinate information in the initial radar detection result into the image coordinate system, obtain the converted radar coordinate information, and determine whether the converted radar coordinate information is in the target pixel area of ​​the target object, wherein the target pixel area of ​​the target object can also be enlarged by a preset multiple, and determine whether the converted radar coordinate information falls into the target pixel area after the enlargement processing, thereby enlarging the target pixel area, which can prevent the radar coordinates after clustering from being outside the target pixel area and missing identification due to calibration error or the center point offset during point cloud clustering; if there are multiple converted radar coordinate information falling into the target pixel area of ​​the same target object at the same time, the corresponding at least two initial radar detection results are determined as the current radar detection results of the target object; Determine whether the current radar detection result of the target object is a correct radar detection result. Specifically, at least two current radar detection results of the target object are screened and processed according to the historical association information of each object, specifically, the current image recognition result of the target object is matched with the historical image recognition results in each historical association information to obtain a first matching result; if the first matching result indicates a successful match, the corresponding historical association information is determined as the historical association information of the target object; the historical radar detection results in the historical association information of the target object are matched with each current radar detection result of the target object to obtain a second matching result; if the second matching result indicates a successful match, the corresponding current radar detection result is determined as the target radar detection result of the target object, and other current radar detection results of the target object that have not been successfully matched are removed.

[0055] After removing the erroneous current radar detection results through historical association information, the object information of the corresponding object is determined based on the target radar detection results of each object and the current image recognition results. In some embodiments, the corresponding motion information can be directly determined from the target radar detection results of each object, and the visual attributes of the corresponding object can be determined according to the current image recognition results of each object. In other embodiments, the target detection device verifies the target radar detection results and the current image recognition results of each object to obtain the target radar detection results after the verification process and the current image recognition results after the verification process; the object information of the corresponding object is determined according to the target radar detection results after the verification process and the current image recognition results after the verification process.

[0056] First, the relationship between the two will be preliminarily determined based on whether the radar coordinate information in the current radar detection result falls within the pixel areas in the current image recognition result, thereby obtaining the current radar detection result and the current image recognition result of each object. After removing the current radar detection result of the false detection and obtaining the target radar detection result, it is necessary to verify the target radar detection result and the current image recognition result of each object to obtain the target radar detection result after verification and the current image recognition result after verification. As an example, after screening the current radar detection result, the target radar detection result after verification and the current image recognition result after verification of each object are determined through matching algorithm processing, and the mapping relationship is as follows:

[0057] in, Indicates the location information of the object. Represents the speed information of the object. It indicates that the radar detection result and the image recognition result are matched and processed to obtain a fusion result; represents the radar object identifier, Indicates radar coordinate information, represents the image object identifier, Indicates the coordinate information of the pixel area of ​​the corresponding object in the current image converted to the coordinate information in the world coordinate system.

[0058]

[0059] in Indicates the position of the radar point in the world coordinate system (the world coordinate system is the same as the radar coordinate system). represents the clustering algorithm for processing radar point cloud data; Represents the position of the image object in the world coordinate system, Represents the calibration function that converts the pixel area coordinate information to the position in the world coordinate system.

[0060] Among them, the steps of the matching algorithm include: constructing an association matrix based on the pixel area coordinate information and radar coordinate information of each object, the association matrix includes matrix elements, and each matrix element represents the distance value between the pixel area coordinate information of an object and the radar coordinate information of the object or other objects; selecting matrix elements from the association matrix according to a preset combination method for combination processing to obtain at least one element combination; determining a target element combination from at least one element combination according to the distance of each matrix element in each element combination; and determining the target radar detection result and the current image recognition result of the object corresponding to each matrix element in the target element combination as the target radar detection result after the corresponding object verification processing and the current image recognition result after the verification processing.

[0061] Among them, the target radar detection results are used as rows and the current image recognition results are used as columns to construct an association matrix. Each matrix element in the association matrix represents the distance value between the pixel area coordinate information of an object and the radar coordinate information of the object or other objects, which is used to indicate the degree of association between the current image recognition result and the target radar detection result. When calculating the distance value, it is necessary to convert the pixel area coordinate information of each object into the world coordinate system, obtain the target image coordinate information of each object, and then calculate the distance value between the target image coordinate information and the radar coordinate information. As an example, the association matrix is ​​as follows:

[0062] In this correlation matrix, Indicates the degree of association between the i-th radar object and the j-th image object in the current frame. The degree of association can be calculated by using the distance value. The distance formula between image object i and radar object j is:

[0063] in, Represents the target image coordinate information, Indicates radar coordinate information, or radar coordinate information after device altitude correction.

[0064] The target detection device selects matrix elements from the correlation matrix according to a preset combination method for combination processing to obtain at least one element combination. The preset combination method can be different combinations between each current image recognition result and different target radar detection results. For any element combination, there is a distance sum. For example, an element combination is element W11, element W 22, element W 33 and element W 44, that is, and , and , and , and , the distance and calculation formula satisfy the following formula:

[0065] in, Represents the distance sum of the element combination. When the distance sum is the smallest, the corresponding element combination is considered to be the best element combination, where the minimum distance and matching of the association matrix can be the KM algorithm (Kuhn-Munkres algorithm).

[0066] In order to prevent the current image recognition result with a long distance from being associated with the target radar detection result, a preset distance threshold can also be set. Specifically, in response to the distance value corresponding to a matrix element in the target element combination being greater than the preset distance threshold, the target detection device selects an element combination from other element combinations according to the distances corresponding to other element combinations to replace the target element combination. The element combination with the smallest distance sum among other element combinations can be selected for replacement. Exemplarily, it is expressed as:

[0067] in, represents the matrix elements, Indicates the preset distance threshold, Indicates a maximum value. If the distance value corresponding to a matrix element is greater than the preset distance threshold, it is set as a maximum value. If the element combination with the smallest distance sum includes the matrix element corresponding to the maximum value, the element combination is eliminated, and the element combination with the smallest distance sum is reselected from the remaining element combinations until there is no maximum value in the selected element combination, which is then determined as the final target element combination.

[0068] In order to elaborate on the target detection method of the present application, Figure 7 The flowchart shown further illustrates it, and the details are as follows: After the program starts, each threshold is initialized and set first; then the internal and external parameters / four-point marking of the camera are performed to obtain the calibration matrix for the conversion between the image coordinate system and the radar coordinate system; image recognition is performed on the preset detection area, and artificial intelligence methods such as deep learning are used to detect the number, coordinates, object type, etc. of objects in the image, and a unique image object identifier is assigned to each object; continuous object detection is adopted so that the same object has a stable image object identifier; the pixel area coordinate information of the object in the current image is converted to the radar coordinate system. In order to simplify the calculation, the coordinate information of the center point of the pixel area can be used as the pixel area coordinate information, and the pixel area coordinate information is converted to the radar coordinate system through the calibration matrix; radar detection is performed on the preset detection area to obtain radar point cloud data, and the radar point cloud data is clustered to obtain the initial radar detection result. The initial radar The detection results include radar detection results of real objects and radar detection results of falsely detected objects; the radar coordinate information in the initial radar detection results is converted to the image coordinate system according to the calibration matrix; time synchronization, based on the timestamp, the historical association pairs of each object at different historical time points are obtained; the historical association information of each object is determined according to the number of historical time points at which each historical association pair appears; the radar splitting points of the large vehicle type are identified according to the historical association information, that is, the current radar detection results of the large vehicle type of the object type are screened and processed to remove the radar splitting points; after removing the radar splitting points, the remaining target radar detection results and the current image recognition results are matched and processed to construct an association matrix, and the association matrix is ​​solved by the KM algorithm to obtain three association results, namely successful association, unassociated target radar detection results, and unassociated current image recognition results; the object information of each object is determined based on the association results.

[0069] See also Figure 8 , Figure 8 It is a structural diagram of an exemplary embodiment of a target detection device shown in the present application. The target detection device 800 includes a detection module 810, a screening module 820 and a determination module 830. The detection module 810 is used to perform radar detection and image recognition on a preset detection area respectively, and obtain the current image recognition result and the current radar detection result of each object in the preset detection area; the screening module 820 is used to respond to the number of current radar detection results of an object being at least two, and to screen and process at least two current radar detection results of the object according to the historical association information of each object, and obtain the target radar detection result of the object, wherein the historical association information includes the correspondence between the historical image recognition result and the historical radar detection result of the object; the determination module 830 is used to determine the object information of the corresponding object based on the target radar detection result of each object and the current image recognition result.

[0070] In the above scheme, the target detection device performs radar detection and image recognition on the preset detection area respectively, and obtains the current image recognition result and current radar detection result of each object in the preset detection area; in response to the number of current radar detection results of the object being at least two, at least two current radar detection results of the object are screened and processed according to the historical association information of the object to obtain the target radar detection result of the object, and the historical association information includes the correspondence between the historical image recognition result and the historical radar detection result of the object; the object information of the object is determined based on the target radar detection result and the current image recognition result of the object. In this way, redundant current radar detection results are screened by historical association information to solve the problem of radar splitting, and the object information of the object is obtained from multiple aspects in combination with the image recognition result, thereby improving the comprehensiveness and accuracy of the object information.

[0071] Among them, the functions of each module can be found in the target detection method embodiment, and will not be repeated here.

[0072] In order to implement the target detection method of the above embodiment, the present application proposes another electronic device, which is specifically referred to as Fig. 9 , Fig. 9 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0073] The electronic device 900 includes a memory 910 and a processor 920 , wherein the memory 910 and the processor 920 are coupled.

[0074] The memory 910 is used to store program data, and the processor 920 is used to execute the program data to implement the target detection method of the above embodiment.

[0075] In this embodiment, the processor 920 may also be referred to as a CPU (Central Processing Unit). The processor 920 may be an integrated circuit chip having signal processing capabilities. The processor 920 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 920 may also be any conventional processor, etc.

[0076] The present application also provides a computer-readable storage medium, such as Fig.10 As shown, the computer-readable storage medium 1000 is used to store program data 1001. When the program data 1001 is executed by the processor, it is used to implement the target detection method in the method embodiment of the present application.

[0077] The method involved in the target detection method embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0078] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target detection method, characterized in that: The target detection method comprises: Performing radar detection and image recognition on the preset detection areas respectively, and obtaining current image recognition results and current radar detection results of each object in the preset detection areas; In response to the number of current radar detection results of an object being at least two, filtering and processing the at least two current radar detection results of the object according to historical association information of each object to obtain a target radar detection result of the object, wherein the historical association information includes a correspondence between historical image recognition results and historical radar detection results of the object; The object information of the corresponding object is determined based on the target radar detection result of each object and the current image recognition result.

2. The target detection method according to claim 1, characterized in that: The step of filtering and processing the at least two current radar detection results of the object according to the historical association information of each object in response to the number of current radar detection results of an object being at least two to obtain the target radar detection result of the object includes: Matching the current image recognition result of the object with the historical image recognition results in each historical association information to obtain a first matching result; In response to the first matching result indicating a successful match, determining the corresponding historical association information as the historical association information of the object; Matching the historical radar detection results in the historical association information of the object with the current radar detection results of the object to obtain a second matching result; In response to the second matching result indicating a successful match, the corresponding current radar detection result is determined as the target radar detection result of the object.

3. The target detection method according to claim 1, characterized in that: The current image recognition result includes a target pixel area of ​​each object in the current image, and the steps of performing radar detection and image recognition on the preset detection area respectively to obtain the current image recognition result and the current radar detection result of each object in the preset detection area include: Performing object recognition processing according to the acquired current image to obtain a target pixel area of ​​each object in the current image; Performing radar detection processing on the preset detection area to obtain an initial radar detection result; Converting radar coordinate information in each initial radar detection result into an image coordinate system of the current image to obtain converted radar coordinate information; In response to at least two converted radar coordinate information being located in the same target pixel area, the corresponding at least two initial radar detection results are determined as current radar detection results of the object corresponding to the target pixel area.

4. The target detection method according to claim 3, characterized in that: The step of performing object recognition processing according to the acquired current image to obtain a target pixel area of ​​each object in the current image includes: Obtaining an initial pixel area of ​​each object in the preset detection area in the current image; In response to the area of ​​the initial pixel region of the object in the current image being greater than a preset area threshold, the corresponding initial pixel region is used as the target pixel region.

5. The target detection method according to claim 1, characterized in that: Before the step of, in response to the existence of at least two current radar detection results of an object, screening and processing the at least two current radar detection results of the object according to the historical association information of each object to obtain the target radar detection result of the object, the method further includes: Obtain the number of historical time points corresponding to the target historical image recognition results and the target historical radar detection results from the historical image recognition results and the historical radar detection results of the same object at each historical time point; In response to the number of the historical time points being greater than a preset threshold, the corresponding relationship between the target historical image recognition result and the target historical radar detection result is determined as the historical association information of the corresponding object.

6. The target detection method according to claim 1, characterized in that: The step of determining the object information of the corresponding object based on the target radar detection result of each object and the current image recognition result comprises: Perform verification processing on the target radar detection result and the current image recognition result of each object to obtain the target radar detection result after verification processing and the current image recognition result after verification processing of each object; The object information of the corresponding object is determined according to the target radar detection result after the verification processing of each object and the current image recognition result after the verification processing.

7. The target detection method according to claim 6, characterized in that: The current image recognition result includes pixel area coordinate information of the object in the current image, the target radar detection result of each object includes radar coordinate information, and the step of performing verification processing on the target radar detection result of each object and the current image recognition result to obtain the target radar detection result of each object after verification processing and the current image recognition result after verification processing includes: Building a correlation matrix based on the pixel region coordinate information and radar coordinate information of each object, the correlation matrix including matrix elements, each matrix element representing a distance value between the pixel region coordinate information of an object and the radar coordinate information of the object or other objects; Selecting matrix elements from the association matrix according to a preset combination method for combination processing to obtain at least one element combination; determining a target element combination from the at least one element combination according to the distances of the matrix elements in the element combinations; The target radar detection result and the current image recognition result of the object corresponding to each matrix element in the target element combination are determined as the target radar detection result after verification processing of the corresponding object and the current image recognition result after verification processing.

8. The target detection method according to claim 7, characterized in that: After the step of determining the target element combination from the at least one element combination according to the distances of the matrix elements in the element combinations, the method further comprises: In response to a distance value corresponding to a matrix element in the target element combination being greater than a preset distance threshold, an element combination is selected from other element combinations according to the distances corresponding to other element combinations and is replaced as the target element combination.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 8.

Images