Target detection method, medium, program product, and electronic device

Abstract

The application relates to the field of electronic equipment, in particular to a target detection method, a medium, a program product and an electronic device. The target detection method comprises the following steps: acquiring an image frame of a target scene collected at a first time, and determining that an overexposure area exists in the image frame; determining a target area corresponding to the overexposure area in the image frame in a radar point cloud frame of the target scene collected at the first time; in the case that it is detected that at least two radar detection objects exist in the target area, selecting a target radar detection object satisfying a recall condition from the at least two radar detection objects; and adding the target radar detection object into a first target detection set, wherein the first target detection set comprises image detection objects determined based on the image frame. The target detection method provided by the application avoids that the radar detection object is mis-recalled due to being shielded, and improves the accuracy of the target detection method.

Description

Target detection methods, media, program products and electronic equipment Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to a target detection method, medium, program product, and electronic equipment. Background Technology

[0002] In the field of driver assistance systems, driver assistance models typically rely on image data acquired during vehicle operation to perceive their environment. However, in brightly lit scenes (such as backlit environments), overexposed areas can appear in the images captured by image acquisition devices. These overexposed areas can obscure important road users (such as vehicles or pedestrians), preventing the driver assistance model from properly perceiving the vehicle's driving environment.

[0003] Therefore, in some technical solutions, radar systems can be used to assist in the perception of the driving environment when a vehicle is driving in a brightly lit scene. Since the radar perception process is not affected by strong light, the driving environment data collected by the radar system can assist the driving model in perceiving or recalling objects covered by overexposed areas.

[0004] In existing recall technologies, the occlusion status of objects detected by radar-assisted sensing is typically not assessed. Therefore, during the recall process, it is inevitable that some occluded objects in overexposed areas will be falsely recalled. Since these occluded objects cannot be detected by machine vision, recalling occluded objects reduces the accuracy of the recall process. Summary of the Invention

[0005] This application provides a target detection method, medium, program product, and electronic device. The following describes this application from multiple aspects, and the embodiments and beneficial effects of these aspects can be referenced each other.

[0006] In a first aspect, this application provides a target detection method, the method comprising: acquiring an image frame of a target scene collected at a first moment, and determining that there is an overexposed area in the image frame; determining a target area in a radar point cloud frame of the target scene collected at the first moment that corresponds to the overexposed area in the image frame; when at least two radar detection objects are detected in the target area, selecting a target radar detection object that meets the recall condition from the at least two radar detection objects; adding the target radar detection object to a first target detection set, wherein the first target detection set includes image detection objects determined based on the image frame.

[0007] According to an embodiment of this application, a target detection method is provided, which avoids the false recall of occluded radar detection objects by recalling unobstructed radar detection objects in the target area to the first target set determined by the image frame, thereby improving the accuracy of the target detection method.

[0008] In some embodiments, the recall conditions include that the occlusion degree of the radar detectors meets the requirements, the target area is a 3D target area, and the radar detectors in the target area are 3D radar detectors. If at least two radar detectors are detected in the target area, the system selects target radar detectors that meet the recall conditions from the at least two radar detectors. This includes: determining at least two 2D radar detectors corresponding to the at least two 3D radar detectors in the 2D target area after the 3D target area is projected onto 2D space; determining whether there are radar detectors that meet the overlap conditions among the at least two 2D radar detectors; corresponding to the existence of radar detectors that meet the overlap conditions, determining the occluder and the obscured among the radar detectors that meet the overlap conditions; and identifying radar detectors that do not meet the overlap conditions, as well as the occluder, as target radar detectors that meet the occlusion degree requirements.

[0009] In some embodiments, determining whether there are radar detectors that satisfy the overlap condition among at least two 2D radar detectors includes: identifying a first 2D radar detector and a second 2D radar detector with an overlapping region among at least two 2D radar detectors; and determining the first 2D radar detector as a radar detector that satisfies the overlap condition if the area ratio of the overlapping region to the area of ​​the first 2D radar detector is greater than an area threshold.

[0010] In some embodiments, corresponding to the existence of radar detection objects that satisfy the overlap condition, determining the obstructing object and the obstructed object among the radar detection objects that satisfy the overlap condition includes: determining the first depth of field and the second depth of field of the first 2D radar detection object and the second 2D radar detection object; if the first depth of field is less than the second depth of field, determining the first 2D radar detection object as the obstructing object and the second 2D radar detection object as the obstructed object.

[0011] According to an embodiment of this application, a method is provided to determine occluders and occluded objects by using depth of field and overlapping areas, avoiding the complex calculations required to determine occluders and occluded objects in the 3D space of radar point cloud frames.

[0012] In some embodiments, the recall conditions further include at least one of the following: the radar-detected object belongs to a predetermined type; the depth of field of the radar-detected object is less than or equal to a depth threshold.

[0013] According to embodiments of this application, a method for determining target recall objects by using radar-detected object type and / or depth information as recall conditions is also provided. By filtering out unimportant or excessively distant items from the radar-detected objects, the computational efficiency of the target detection method of this application is improved.

[0014] In some embodiments, the predetermined type includes at least one of people and vehicles.

[0015] In some embodiments, acquiring an image frame of the target scene captured at a first moment and determining that there are overexposed areas in the image frame includes: determining that the image frame has M corresponding overexposed areas in M ​​different color spaces, and that there are overlapping areas among the M overexposed areas, where M is an integer greater than 1; and taking the overlapping areas as the overexposed areas of the image frame.

[0016] According to the embodiments of this application, a method for determining overexposed areas using multiple color spaces is provided, which reduces the possibility of misjudging areas such as pure white as overexposed areas and makes the overexposed areas determined from image frames more accurate.

[0017] Secondly, embodiments of this application provide a readable storage medium including one or more programs, which, when executed on an electronic device, cause the electronic device to implement the method of any embodiment of the first aspect.

[0018] Thirdly, this application provides a program product that, when executed on an electronic device, causes the electronic device to implement the method of any embodiment of the first aspect.

[0019] Fourthly, this application provides an electronic device, comprising: at least one memory for storing one or more programs; and at least one processor for executing one or more programs to cause the electronic device to implement the method of any embodiment in the first aspect. Attached Figure Description

[0020] Figure 1 is a schematic diagram of an exemplary target scenario provided in an embodiment of this application;

[0021] Figure 2 is a schematic diagram of the overexposed area in an image frame provided in an embodiment of this application;

[0022] Figure 3 is an exemplary flowchart of the target detection method provided in an embodiment of this application;

[0023] Figure 4 is an exemplary flowchart of the target radar detection method provided in an embodiment of this application;

[0024] Figure 5 is a modular schematic diagram of the target detection system provided in an embodiment of this application;

[0025] Figure 6 is a block diagram of the electronic device provided in an embodiment of this application;

[0026] Figure 7 is a block diagram of a system-on-a-chip provided in an embodiment of this application. Detailed Implementation

[0027] This application provides a target detection method. The method provided by this application can recall objects that were missed in overexposed areas of an image that cannot be identified by machine vision in a strong light scene, thereby improving the visual detection recall rate in overexposed environments.

[0028] For ease of understanding, this application first describes the terms that may be involved in the embodiments.

[0029] 1. Recall rate:

[0030] In some embodiments, recall is a metric used to represent the completeness of an algorithm's detection capabilities. For example, 10,000 CT images (representing all samples) can be input into a deep learning model to predict 1,000 positive CT images (representing positive samples). Assuming the output includes 873 correctly detected positive computed tomography (CT) images and 9,000 correctly detected negative CT images, then when no negative CT images are detected as positive, the deep learning model will incorrectly predict a total of 10,000 - 9,000 - 873 = 127 positive CT images as negative CT images. In this case, the total recall is 873 / 1000 = 87.3%. That is, the higher the recall, the stronger the model's ability to detect positive samples, and the lower the likelihood of missed detections.

[0031] 2. Target Recall Methods:

[0032] In some embodiments, the targeted recall method is a method for improving the recall rate of positive samples. The targeted recall method can re-predict positive samples that were incorrectly predicted as negative samples by the algorithm, and recall these positive samples back to the set of already predicted positive samples, thereby increasing the proportion of correctly predicted positive samples among all positive samples, i.e., improving the recall rate.

[0033] 3. Overexposed areas:

[0034] In some embodiments of this application, the formation of overexposed areas is related to strong lighting. Due to excessively strong lighting, the brightness values ​​of pixels acquired by the image acquisition device are set to the maximum values ​​in the corresponding color space. For example, in the Red Green Blue (RGB) color space, the values ​​of the Red Channel (R), Green Channel (G), and Blue Channel (B) are all set to 255. In the image, overexposed areas typically exist in the form of bright spots, and for the objects included in these overexposed areas, their features, textures, and colors cannot be detected due to overexposure.

[0035] Below, we will first introduce the impact of strong light scenes on target detection by electronic devices.

[0036] Figure 1 illustrates an exemplary target scenario, specifically a vehicle driving on an urban road. Vehicles 110 and 120 are traveling on a highway, with vehicle 120 positioned to the left front of vehicle 110. A traffic cone 130 is placed on the highway in the blind spot of vehicle 110, which is obscured by vehicle 120. It is understood that vehicle 110 can drive on the highway using an assisted driving system. To perceive the driving environment of vehicle 110, the image acquisition device 111 and radar 112 on vehicle 110 can transmit image data acquired at various times to the assisted driving system, enabling the system to perceive the driving scene based on images or radar point cloud data at any time.

[0037] When the light intensity of the driving scene shown in Figure 1 is too strong, that is, when the driving scene is a strongly lit scene, large overexposed areas will appear in the image captured by the image acquisition device 111. As shown in Figure 2, Figure 2 shows a schematic diagram of the overexposed areas in the image frame. Specifically, Figure 2 shows the overexposed image captured by the image acquisition device 111 from a frontal view in the driving scene shown in Figure 1. In Figure 2, due to the strong light intensity of the driving scene, there are two overexposed areas in the image captured by the image acquisition device 111: the first overexposed area 210 on the left side of Figure 2 and the second overexposed area 220 in the middle.

[0038] As shown in Figure 1, the first overexposed area 210 obscures the vehicle 120 in Figure 1. At this time, according to machine vision, the driver assistance system can no longer identify the vehicle 120 in the driving scene based on the image.

[0039] In other words, under strong light, machine vision alone may miss vehicles 120 in the first overexposed area 210 caused by strong light, resulting in a low recall rate.

[0040] As mentioned earlier, since machine vision cannot effectively identify objects in overexposed areas of an image, it is possible to acquire additional data from sensors that are not sensitive to light intensity, in addition to machine vision, and use this data to assist in identifying objects obscured by overexposed areas.

[0041] In some embodiments, this data collected from different sensors is referred to as multimodal data.

[0042] In some embodiments, multimodal data may include image data and radar point cloud data. Radar (e.g., lidar or millimeter-wave radar) is less affected by environmental visibility and has a strong ability to identify objects in harsh environments (e.g., rainy nights, dense fog, strong light, etc.), and can serve as a supplement to visual data in harsh environments.

[0043] For example, the driver assistance system can use radar point cloud data collected by radar 112 to identify objects in the first overexposed area 210. By recalling the objects identified by the radar point cloud data to the set of objects identified by the image data, the missed objects can be recalled.

[0044] However, the above solution indiscriminately recalls objects obscured by overexposed areas, regardless of whether the object can be recognized by machine vision under normal conditions. For example, in the above technical solution, radar 112 can identify multiple objects in the target scene obscured by the first overexposed area 210, including vehicle 120 and cone 130. However, there is also an obscuration relationship between vehicle 120 and cone 130 in the first overexposed area 210. That is, even if the first overexposed area 210 does not exist, cone 130 cannot be captured by the image acquisition device due to the obstruction of vehicle 120. In other words, cone 130 cannot appear in the image acquired by the image acquisition device and thus cannot be recognized by machine vision.

[0045] As mentioned above, recalling all objects in the first overexposed area 210 without considering the occlusion relationship between objects in the occlusion area would lead to the wrong recall of objects that should not be recalled, thus reducing the accuracy of the recall.

[0046] Therefore, this application proposes a target detection method. The method includes: acquiring an image frame of a target scene collected at a first moment, and determining that an overexposed area exists in the image frame; determining a target area in a radar point cloud frame of the target scene collected at the first moment that corresponds to the overexposed area in the image frame; if at least two radar detection objects are detected in the target area, selecting a target radar detection object that meets the recall condition from the at least two radar detection objects; adding the target radar detection object to a first target detection set, wherein the first target detection set includes image detection objects determined based on the image frame.

[0047] In the target recall method of this application, when recalling objects in the overexposed area of ​​an image frame based on radar detection results, only the radar-detected targets that are not obscured in the radar point cloud frame are recalled, thereby improving the accuracy of target detection.

[0048] It is understood that in some embodiments of this application, an overexposed area may refer to a set of pixels that are overexposed in an image frame; a target area may refer to a set of point clouds in a radar point cloud frame that corresponds to the overexposed area in the image frame.

[0049] In addition, in some embodiments, the radar detection object can be an object present in the radar point cloud frame detected by target detection of the radar point cloud frame; the target radar detection object can be a radar detection object located in the target area and meeting the recall conditions.

[0050] It is understood that in some embodiments, recall conditions refer to the judgment conditions used to determine whether a radar detection object needs to be recalled. For example, the recall condition may be that the radar detection object is not obstructed. For two radar detection objects that have an obstruction relationship, the two radar detection objects can be distinguished into an obstructing object and an obstructed object according to the obstruction relationship. The obstructed object refers to the radar detection object that is obstructed by the obstructing object. For the obstructed object, even if there is no overexposed area in the image, it cannot be identified by machine vision because it is obstructed.

[0051] It is understood that in some embodiments of this application, the image detection objects can be objects included in the target scene determined from the image frame by a machine vision recognition method; the first target detection set can be a set composed of image detection objects.

[0052] According to the above scheme, when the driver assistance system detects an overexposed area in the image data, it can execute a target recall method to improve the recall rate of machine vision recognition.

[0053] For example, in some embodiments, an electronic device (e.g., a vehicle-mounted system) can first acquire image frames and radar point cloud frame data of the target scene based on the image acquisition device 111 and radar 112 in the vehicle 110, where the image frame can be shown in Figure 2. Then, the electronic device can determine one or more first overexposed regions 210 included in the image frame based on the image frame. When an object in the target scene is located in the first overexposed region 210, it will be occluded by the first overexposed region 210 and cannot be identified by machine vision methods. Therefore, the objects included in the first overexposed region 210 can be determined by the projection of the objects in the radar point cloud frame into the image frame. Then, by judging the occlusion relationship between the objects in the first overexposed region 210, it can be determined which of the multiple objects projected onto the first overexposed region 210 are not occluded. Finally, after determining the unoccluded objects in the first overexposed region 210, the electronic device can recall these objects and add them to the set of objects identified by the image frame, thus completing the recall of the missed objects.

[0054] The following section will use the vehicle 120 and cone 130 in Figure 1 as examples of objects in an overexposed area, and the vehicle-mounted unit as an electronic device for performing the target detection method, to introduce the technical solution provided in this application.

[0055] Figure 3 is an exemplary flowchart of the target detection method provided in an embodiment of this application. Referring to Figure 3, the flow of the target detection method includes:

[0056] S100: Acquire the image frame of the target scene captured at the first moment.

[0057] In some embodiments, the first moment can be any moment when an image frame is captured.

[0058] In some embodiments, the target scenario may include real-world scenarios in which the vehicle 110 is actually driving, such as different scenarios like city roads, highways, tunnels, underground parking lots, port container yards, bridges, or viaducts, which are not limited here.

[0059] In some embodiments, the image frame of the target scene can be a static image at a first moment or an image corresponding to the video at the first moment. The static image or the image frame of the video at the first moment of the target scene can be captured by an image acquisition device 111 (e.g., a multi-angle camera) installed on the vehicle 110. Furthermore, the vehicle's infotainment system can obtain the image frame data captured by the image acquisition device 111 on the vehicle 110 via the data bus.

[0060] S110: Determine whether there is an overexposed area in the image frame.

[0061] If the judgment result of S110 is yes, then S120 is executed to determine the target area corresponding to the overexposed area in the radar point cloud frame collected at the first moment; if the judgment result is no, it means that there is no need to recall, and S190 is executed directly to obtain the target detection result based on the image frame.

[0062] It is understood that in some embodiments, since the target scene may be a non-strong light scene, the obtained image frame may not be overexposed. Therefore, it can be determined whether the image frame has been overexposed by judging whether there is an overexposed area in the image frame.

[0063] In some embodiments, overexposed areas determined using a single color space are prone to errors, such as misclassifying white areas as overexposed. Therefore, some embodiments of this application propose an implementation method that combines multiple color spaces for overexposed area determination. Furthermore, it is understood that embodiments of this application may also employ other existing technical solutions to determine whether overexposed areas exist in an image frame.

[0064] In some embodiments, the method for determining the overexposed regions in each color space includes: determining that the image frame has M corresponding overexposed regions in M ​​different color spaces, and that there is an overlapping region among the M overexposed regions, where M is an integer greater than 1.

[0065] It is understandable that the coverage area of ​​the overexposed region determined according to different color spaces in an image frame may be different. Therefore, the overlapping area can be used as the overexposed region of the image frame to improve the accuracy of overexposed region determination.

[0066] In some embodiments, color spaces can be represented in various different forms, such as the HSV color space represented by three channels: Hue (H), Saturation (S), and Value (V); the RGB color space represented by three channels: Red (R), Green (G), and Blue (B); and the LAB color space represented by a Lightness (L), A-axis (A), and B-axis (B).

[0067] It is understood that the three color spaces shown above are only one combination selected when M=3 to determine the overexposed areas in each color space. The value of M can also be different values ​​such as 4, 5, 10, or 20. This application does not limit the specific value of M or the specific method of selecting the M color spaces.

[0068] For ease of description, the following text will take M=3 and the combination of color spaces as HSV color space, RGB color space and LAB color space as an example to introduce the method for determining the M overexposure areas corresponding to M different color spaces.

[0069] It is understandable that brightness is represented differently in different color spaces. For example, in the HSV color space, the V component represents the brightness of the corresponding pixel. Thus, pixels with a V component greater than an overexposure threshold can be identified as overexposed pixels. For example, V > 250 can be used as the overexposure threshold for determining whether a pixel is overexposed. The specific value of the overexposure threshold is not limited in this application.

[0070] In some embodiments, after determining the overexposed areas in each color space, the overlapping area of ​​the overexposed areas in each color space can be used as the overexposed area of ​​the image frame. For example, in the HSV color space, RGB color space, and LAB color space described above, the overexposed areas in the above three spaces are determined respectively. Then, an intersection operation is performed on the overexposed areas in the above three color spaces to obtain the intersection area of ​​the overexposed areas in the three color spaces, and this intersection area is determined as the overexposed area of ​​the image frame.

[0071] S120: Acquire the radar point cloud frame of the target scene acquired at the first moment, and determine the target area in the radar point cloud frame of the target scene acquired at the first moment that corresponds to the overexposed area in the image frame.

[0072] It is understandable that, after determining that there are overexposed areas in the image frame, in order to determine the area in the radar point cloud frame that represents the overexposed area, the corresponding target area in the radar point cloud frame can be determined based on the overexposed area in the image frame.

[0073] In some embodiments, the radar point cloud frame of the target scene can be radar point cloud data collected by radar 112 (e.g., millimeter-wave radar) installed on vehicle 110 at a first moment. Furthermore, the vehicle's infotainment system can obtain the radar point cloud frame data collected by radar 112 on vehicle 110 via a data bus.

[0074] In some embodiments, after acquiring the radar point cloud frame as described above, the target region corresponding to the overexposed area in the radar point cloud frame can be determined by projection. For example, the overexposed area in the image frame (e.g., the first overexposed area 210 or the second overexposed area 220) can be projected onto the radar point cloud frame to determine the target region in the radar point cloud frame that corresponds to the overexposed area in the image frame. It can be understood that any point in the radar point cloud within the target region obtained by the above method has its corresponding position in the image frame within the overexposed area of ​​the image frame.

[0075] It is understood that, in order to improve the accuracy of the recall, in this embodiment of the application, target recall needs to be based on image frames and radar point cloud frames acquired at the same time (i.e., the first time). The same time here refers to two acquisition times with a time difference less than a time threshold. For example, the time difference between the acquisition times of the image frame and the radar point cloud frame is less than 1ms, 2ms, 5ms, 10ms, or 50ms. This application does not impose any restrictions on this.

[0076] S130: Determine whether the number of radar-detected objects in the target area is at least two.

[0077] It is understood that the number of 3D detection frames used to identify radar detection objects in the target area determined by the above method can include 0, 1, or at least two. An obstruction is only possible when there are at least two radar detection objects in the target area of ​​the radar point cloud frame. Obstructions do not require recall. Therefore, it can be determined here whether the number of radar detection objects in the target area is greater than 1. If the determination result is yes, then S140 is executed to identify radar detection objects that meet the recall conditions; if the determination result is no, then S170 is executed to further determine the number of radar detection objects.

[0078] It is understandable that the target area determined by S120 can be represented in 3D (3-Dimension, 3D) space. Therefore, the target area in the radar point cloud frame can be a 3D target area, and the radar detection objects in the 3D target area can also include 3D radar detection objects.

[0079] In some embodiments, 3D radar objects can be detected using a target detection model, which may include a machine learning model, such as PointNet, for detecting radar objects present in a radar point cloud frame. The target detection model can output the detected 3D radar objects, which can be represented by a 3D bounding box (3DBBOX).

[0080] For example, a radar point cloud frame can be input into a point network to obtain a 3D detection box used to identify 3D radar detection objects present in the radar point cloud frame.

[0081] S140: Identify target radar detectors that meet the recall criteria.

[0082] In some embodiments of this application, the recall conditions include that the degree of obstruction of the radar-detected object meets the requirements, that is, the radar-detected object is not obstructed.

[0083] It is understandable that S130 can identify multiple 3D radar detection objects in the target area. However, if one 3D radar detection object is obscured by other 3D radar detection objects, even without overexposure, the obscured 3D radar detection object cannot be identified by machine vision. Therefore, the target radar detection object can be determined by judging whether each 3D radar detection object meets the recall criteria.

[0084] It is understood that, in some embodiments, the method for determining the target radar detection object includes: determining at least two 2D radar detection objects corresponding to at least two 3D radar detection objects in the 2D target area after the 3D target area is projected onto the 2D space; determining whether there are radar detection objects that satisfy the overlap condition among the at least two 2D radar detection objects; corresponding to the existence of radar detection objects that satisfy the overlap condition, determining the obstructing objects and the obstructed objects among the radar detection objects that satisfy the overlap condition; and determining the radar detection objects that do not satisfy the overlap condition and the obstructing objects as target radar detection objects whose obstruction degree meets the requirements.

[0085] The following section will detail the specific methods for determining whether a target detected by radar meets the recall criteria; these will not be elaborated upon here.

[0086] S150: Add the target detected by the radar to the first target detection set to obtain the target detection result of the image frame.

[0087] It is understandable that the target radar detection objects are those that the machine vision target detection model misses due to overexposure. The missed target radar detection objects can be recalled to improve the recall rate of target detection by machine vision-based target detection models in overexposure scenes.

[0088] In some embodiments, the first object detection set is a set of image detection objects determined by an object detection model, wherein the image detection objects can be objects present in an image frame as determined by object detection of the image frame using a machine vision-based object detection model, such as RetinaNet.

[0089] In some embodiments, the target radar detectors added to the first target set are objects that were missed in the image detectors determined according to recall conditions.

[0090] For example, RetinaNet can be used to perform target detection on image frames, resulting in a first target detection set that includes the detected objects in the image. Then, according to the scheme described in S110 to S140 above, the target radar detection objects to be recalled are determined. In this way, the target radar detection objects can be added to the first target detection set, avoiding the problem of missed detections in the machine vision-based first target detection set caused by overexposed image frames.

[0091] S160: Output target detection results.

[0092] S170: Determine whether the number of radar-detected objects in the target area is one.

[0093] It is understandable that if the determination in S130 is negative, the number of radar detectors in the target area can be less than two. If the determination in S170 is positive, meaning that there is only one radar detector in the target area, there is no possibility of other radar detectors causing obstruction. Therefore, S180 can be executed to identify that radar detector as the target radar detector.

[0094] If the judgment in S170 is negative, that is, there is no radar detection object in the target area, there is no radar detection object to be recalled at this time, so S190 can be executed to determine the target detection result at the first moment.

[0095] S180: Identify radar detection objects with a quantity equal to 1 as target radar detection objects.

[0096] According to the judgment result of S170, the target area contains only one unobstructed radar detection object, which can be directly identified as the target radar detection object.

[0097] S190: Based on the image detection objects determined in the image frame, a first target detection set is obtained, and the first target detection set is used as the target detection result of the image frame.

[0098] Based on the judgment results of S110 or S170, when there is no overexposed area in the image frame or no radar detection object in the target area corresponding to the overexposed area, there is no object to be recalled. Therefore, the first target detection set composed of the image detection objects determined according to the image frame can be updated, and the first target detection set can be directly used as the target detection result of the image frame at the first moment.

[0099] In other embodiments of this application, after determining the overexposed area corresponding to the image frame in S110, the data quality of the overexposed area can be further improved to enhance the recall accuracy of the image detection method.

[0100] For example, in some embodiments, since a large number of overexposed areas consist of only one or two pixels, these few overexposed pixels appear as salt-and-pepper noise in the image frame. Salt-and-pepper noise refers to pixels in an image frame whose brightness deviates significantly from that of their neighboring pixels. Because of the extreme brightness contrast between these pixels and their surroundings, problems such as missed detections can occur during target recognition in machine vision.

[0101] Therefore, to reduce the occurrence of the above problems, filtering algorithms can be used to remove salt-and-pepper noise from image frames. A filtering algorithm is an algorithm that removes interference from noisy data according to certain rules.

[0102] For example, to remove salt-and-pepper noise, a median filtering algorithm can be used to remove pixels in the image frame whose brightness deviates excessively from the surrounding pixels, thereby achieving the purpose of removing salt-and-pepper noise from the image frame. The specific filtering algorithm is not limited herein.

[0103] In some embodiments, the boundaries of overexposed areas in an image frame are not smooth enough. Therefore, an opening operation can be performed on the image frame using an elliptical structuring element to smooth the boundaries of overexposed areas. The opening operation removes smaller burrs than the elliptical structuring element from the edges of the overexposed areas by first eroding and then dilating.

[0104] In some embodiments, only large overexposed areas affect object recognition in an image. That is, only large overexposed areas affect the recall rate based on image frames. Therefore, it is also necessary to filter out overexposed areas with an area smaller than a preset threshold. For example, the ratio of the overexposed area to the image frame area can be used as a criterion. When this ratio is less than the preset threshold, smaller overexposed areas are removed from the set of overexposed areas.

[0105] In some embodiments, the preset threshold can be set to values ​​such as 1%, 2% or 3%, and this application does not limit it.

[0106] In some embodiments, the specific method for determining the target radar detection object that meets the recall conditions can be illustrated by Figure 4.

[0107] Referring to Figure 4, the flow of the target radar detection method may include:

[0108] S141: Determine at least two 2D radar detection objects corresponding to at least two of the 3D radar detection objects in the 2D target area after the 3D target area is projected onto the 2D space.

[0109] In some embodiments, the projection method of projecting a 3D target area onto 2D space may include bird's-eye view (BEV) projection. When a 3D radar detection object is located within a 3D target area, the 3D detection frame of that 3D radar detection object can also be projected onto the 2D target area to obtain a 2D radar detection object.

[0110] In some embodiments, a 2D radar detection object can be obtained by projecting a 3D radar detection object onto a 2D space. The 2D radar detection object can be represented by a 2D detection box (2DBBOX).

[0111] In some embodiments, whether a 2D radar detection object is located within a 2D target region can be determined by the percentage of the 2D target region that covers the 2D detection frame. Specifically, if the percentage is greater than a coverage threshold, it indicates that the area of ​​the 2D radar detection object obscured by the 2D target region is too large to be detected by a machine vision-based detection model. In this case, the radar detection objects corresponding to these 2D radar detection objects that cannot be detected by the machine vision detection model can be identified as 3D radar detection objects present in the 3D target region. The coverage threshold can be set to 70%, 80%, or 90%, which is not limited herein.

[0112] For example, radar point cloud frames can be projected into 2D space using a bird's-eye view projection method to obtain a 2D target region corresponding to a 3D target region and a 2D detection box corresponding to a 3D detection box. Then, the area ratio of the 2D target region within the coverage area of ​​the 2D detection box is determined. If the ratio is greater than a coverage threshold, the 2D radar detection object corresponding to the 2D detection box can be identified as being located within the 2D target region.

[0113] S142: Determine whether at least two 2D radar detection objects exist that satisfy the overlap condition.

[0114] Understandably, in S140, the recall conditions include meeting the requirements for the degree of obstruction of the radar-detected object.

[0115] In some embodiments, the radar detection object that meets the obstruction requirements as described above is an unobstructed radar detection object. Therefore, whether the obstruction degree of a radar detection object meets the requirements can be determined by whether the radar detection object is obstructed.

[0116] It is understandable that for a radar-detected object that is blocked, there is an overlapping area between its corresponding 2D radar-detected object and other 2D radar-detected objects. Therefore, the overlap condition between the 2D radar-detected objects corresponding to the radar-detected object can be used to determine whether the radar-detected object is blocked, and then determine whether the degree of blockage of the radar-detected object meets the requirements.

[0117] For example, 2D radar detection objects that do not meet the overlap condition, as well as 2D radar detection objects that meet the overlap condition but are obstructions, can be identified as radar detection objects whose obstruction degree meets the requirements.

[0118] Therefore, in order to determine whether the obstruction of radar detection objects meets the requirements, it is first necessary to determine whether there are radar detection objects that satisfy the overlap condition among at least two 2D radar detection objects.

[0119] In some embodiments, the method for determining radar detectors that satisfy the overlap condition includes: identifying a first 2D radar detector and a second 2D radar detector with an overlapping region among at least two 2D radar detectors; and determining the first 2D radar detector as a radar detector that satisfies the overlap condition if the area ratio of the overlapping region to the area of ​​the first 2D radar detector is greater than an area threshold.

[0120] For example, at least two 2D radar detection objects obtained in the first S141 can be traversed, and the traversed 2D radar detection objects can be used as the first 2D radar detection object. By performing an intersection operation between the first 2D radar detection object and the 2D detection frames of any other second 2D radar detection object, the overlapping area of ​​the first 2D radar detection object and the second radar detection object can be obtained. Then, the ratio between the overlapping area and the first 2D radar detection object is calculated. When the ratio is greater than the area threshold, it is determined that the two radar detection objects have overlapped, and the first 2D radar detection object is determined as the radar detection object that meets the overlap condition. This process continues until all 2D radar detection objects have been traversed. The area threshold can be set to 70%, 80%, or 90%, which is not limited in this application.

[0121] S143: For 2D radar detection objects that satisfy the overlap condition, identify the obstructing objects and the obscured objects among the 2D radar detection objects that satisfy the overlap condition.

[0122] It is understandable that satisfying the overlap condition can only determine that there is an overlap between two 2D radar detection objects, but cannot determine which of the two 2D radar detection objects is not obstructed. Therefore, it is impossible to identify the unobstructed 2D radar detection object as the target radar detection object whose obstruction degree meets the requirements.

[0123] Therefore, in order to determine the obstructed 2D radar detection object, it is also necessary to determine the obstructing object and the obstructed object in the two 2D radar detection objects mentioned above.

[0124] In some embodiments, the occluder and the occluded object can be determined by depth of field, and the determination method includes: determining a first depth of field and a second depth of field of the first 2D radar detection object and the second 2D radar detection object; if the first depth of field is less than the second depth of field, determining the first 2D radar detection object as the occluder and the second 2D radar detection object as the occluded object.

[0125] Depth of field can be used to represent the distance information between a radar-detected object and the radar. The greater the depth of field, the farther away the radar-detected object is from the radar. Therefore, for two overlapping radar-detected objects, a first radar-detected object and a second radar-detected object, if the depth of field of the first radar-detected object is less than that of the second radar-detected object, it means that the first radar-detected object is closer to the radar, thus blocking the second radar-detected object.

[0126] For example, among 2D radar detection objects that meet the overlap condition, the first 2D radar detection object has a first depth of field of 10 meters and the second 2D radar detection object has a second depth of field of 20 meters. In this case, the first 2D radar detection object corresponding to the first depth of field with a smaller depth of field is identified as the occluder, and the second 2D radar detection object corresponding to the second depth of field with a larger depth of field is identified as the occluded object.

[0127] S144: Radar detection objects that do not meet the overlap condition, as well as obstructions, are identified as target radar detection objects whose obstruction degree meets the requirements.

[0128] It is understood that, according to the methods in S141 to S142 above, radar detection objects in the target area can be identified as radar detection objects that do not meet the overlap condition, and as obstructing objects and objects that are obstructed among radar detection objects that meet the overlap condition.

[0129] It can be understood that both radar detection objects that do not meet the overlap condition and those that do meet the overlap condition, including any obstructing objects, are considered unobstructed radar detection objects. In other words, the degree of obstruction of obstructing objects in both radar detection objects that do not meet the overlap condition and those that do meet the overlap condition satisfies the requirements.

[0130] Therefore, objects that do not meet the overlap condition and objects that do meet the overlap condition can be identified as target radar objects with sufficient obstruction.

[0131] In some embodiments, when the 2D target area includes at least two radar detectors, the obstruction determined according to the method in S143 can be identified as a target detector because it will not be obstructed. The obstructed object determined according to the method in S143 cannot be identified as a target detector to be recalled because it has already been obstructed by another obstruction. If the radar detectors do not meet the overlap condition, these radar detectors will not be obstructed by other radar detectors and can therefore also be identified as target radar detectors.

[0132] In other embodiments of this application, the recall conditions, in addition to the overlapping conditions, may include at least one of the following: the radar-detected object belongs to a predetermined type; the depth of field of the radar-detected object is less than or equal to a depth threshold.

[0133] In some embodiments, the predetermined type may include at least one of people and vehicles. It is understood that people and vehicles are of greater importance than other types of radar detectors because they are related to the life safety of drivers and pedestrians. Thus, 2D radar detectors whose semantic information does not belong to the predetermined type can be removed to eliminate interference from non-predetermined type radar detectors on 2D radar detectors of the people and vehicle types during the recall process.

[0134] It is understood that people and vehicles are only one exemplary preset type in the embodiments of this application. Those skilled in the art can also set preset types according to actual business needs, such as roadblocks, rolling stones or buildings. This application does not limit them here.

[0135] It is understood that in some embodiments, radar detectors located further away from vehicle 110 may also be included in the radar detectors. These radar detectors located further away from vehicle 110 pose a lower threat to vehicle 110 performing the target detection method of this application. Thus, radar detectors located further away from vehicle 110 can be filtered out from the radar detectors to improve the efficiency of recall.

[0136] This application also provides a target detection system using the image detection method of this application. Specifically, Figure 5 shows a modular schematic diagram of the target detection system 300, which may include multiple functional modules, including: an overexposed area detection module 310, a radar point cloud target detection module 320, a target radar detection object determination module 330, and a recall module 340. The data input to the target detection system 300 includes image frame 301 and radar point cloud frame 302. The specific workflow of the target detection system 300 is described below.

[0137] The specific workflow of the target detection system 300 can be understood as follows: Image frames 301 and radar point cloud frames 302 from the target scene are input into the target detection system 300, so that the overexposed area detection module 310 acquires image frame 301 and the radar point cloud target detection module 320 acquires radar point cloud frame 302; the overexposed area detection module 310 can determine the overexposed area 311 from image frame 301; the radar point cloud target detection module can determine the radar detection object 321 included in the radar point cloud frame from radar point cloud frame 302; the overexposed area 311 and radar detection object 321 are input together into the target radar detection object determination module 330, so that the target radar detection object determination module 330 determines the target radar detection object 331 located in the overexposed area 311 from the radar detection object 321; the recall module 340 can recall the target radar detection object 331 back to the set of objects determined based on the image frames based on the point cloud information of the target radar detection object 331.

[0138] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any storage medium (e.g., magnetic medium, optical medium, semiconductor medium, etc.) capable of storing and / or retrieving data by a computing device. The computer-readable storage medium includes instructions that direct a computing device to implement the target detection method provided in this application.

[0139] This application also provides a program product, which may be a software or program product including instructions capable of running on a computing device or stored on any available medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to implement the target detection method provided in this application.

[0140] This application also provides an electronic device, including one or more processors and one or more memories, wherein the one or more processors can be used to execute instructions to implement the target detection method provided in this application.

[0141] For example, FIG6 shows a block diagram of an electronic device 400 according to an embodiment of the present application. The electronic device 400 may include one or more processors 401 coupled to a controller hub 403. In at least one embodiment, the controller hub 403 communicates with the processor 401 via a multi-branch bus such as a Front Side Bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or a similar connection. The processor 401 executes instructions controlling general types of data processing operations. In some embodiments, the controller hub 403 may include, but is not limited to, a Graphics & Memory Controller Hub (GMCH) (not shown) and an Input / Output Hub (IOH) (which may be on separate chips) (not shown), wherein the GMCH may include memory and a graphics controller and is coupled to the IOH.

[0142] Electronic device 400 may also include a coprocessor 402 and a memory 404 coupled to a controller hub 403. Alternatively, one or both of the memory and the GMCH may be integrated within the processor (as described in the embodiments of this application), with memory 404 and coprocessor 402 directly coupled to processor 401 and controller hub 403, which is located on a single chip with IOH. Memory 404 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both.

[0143] The memory 404 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of the instructions. The instructions may include instructions that, when executed by at least one processor, cause the electronic device 400 to perform the method shown in FIG3. When the instructions are executed on a computer, they cause the computer to perform the target detection method disclosed in the embodiments of this application.

[0144] In one embodiment, coprocessor 402 is a dedicated processor, such as, for example, a high-throughput many integrated core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on graphics processing units (GPGPU), or an embedded processor, etc. Optional properties of coprocessor 402 are indicated by dashed lines in Figure 6.

[0145] In one embodiment, electronic device 400 may further include a Network Interface Controller (NIC) 406. The network interface 406 may include a transceiver for providing a radio interface for electronic device 400 to communicate with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, the network interface 406 may be integrated with other components of electronic device 400. The network interface 406 can implement the functions of the communication unit in the above embodiments.

[0146] Electronic device 400 may further include input / output (I / O) devices 405. I / O 405 may include: a user interface designed to enable a user to interact with electronic device 400; a peripheral component interface designed to enable peripheral components to also interact with electronic device 400; and / or sensors designed to determine environmental conditions and / or location information related to electronic device 400.

[0147] It is worth noting that Figure 6 is merely exemplary. That is, although Figure 6 shows an electronic device 400 including multiple devices such as a processor 401, a controller hub 403, and a memory 404, in actual applications, devices using the methods of the embodiments of this application may include only a portion of the devices in the electronic device 400. For example, it may only include the processor 401 and the network interface 406. The nature of optional devices in Figure 6 is shown with dashed lines.

[0148] Figure 7 shows a block diagram of a System on Chip (SoC) 500 applied to an electronic device according to an embodiment of this application. In Figure 7, similar components have the same reference numerals. Additionally, dashed boxes represent optional features of more advanced SoCs. In Figure 7, the SoC 500 includes: an interconnect unit 550 coupled to a processor 510; a system proxy unit 580; a bus controller unit 590; an integrated memory controller unit 540; one or more coprocessors 520, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random-access memory (SRAM) unit 530; and a direct memory access (DMA) unit 560. In one embodiment, the coprocessor 520 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a general-purpose computing on graphics processing units (GPGPU), a high-throughput MIC processor, or an embedded processor.

[0149] Static Random Access Memory (SRAM) cell 530 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of those instructions. These instructions may include instructions that, when executed by at least one processor, cause the SoC to implement the methods shown in the embodiments of this application. When the instructions are executed on a computer, they cause the computer to perform the target detection method disclosed in the embodiments of this application.

[0150] It should be noted that the terminology used in the embodiment section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between associated obstacles, indicating that three relationships can exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. In addition, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more.

[0151] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0152] References to "one embodiment" or "some embodiments" as used in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0153] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product may include one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium, etc.

[0154] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium can include various media capable of storing program code, such as read-only memory, random access memory, magnetic disks, or optical disks.

[0155] The above description is merely a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims

1. A target detection method, characterized in that, The method includes: acquiring an image frame of a target scene captured at a first moment, and determining that an overexposed area exists in the image frame; determining a target area in a radar point cloud frame of the target scene captured at the first moment that corresponds to the overexposed area in the image frame, wherein the target area is a 3D target area; when at least two radar detection objects are detected in the target area, selecting a target radar detection object that meets the recall criteria from the at least two radar detection objects, wherein the recall criteria include that the occlusion degree of the radar detection object meets the requirements, and that the radar detection object in the target area is a 3D radar detection object; adding the target radar detection object to a first target detection set, wherein the first target detection set includes targets based on the image... The image detection objects determined by the frame; wherein, when at least two radar detection objects are detected in the target area, selecting target radar detection objects that meet the recall conditions from the at least two radar detection objects includes: determining at least two 2D radar detection objects corresponding to at least two 3D radar detection objects in the 2D target area after the 3D target area is projected onto 2D space; determining whether there are radar detection objects that meet the overlap conditions among the at least two 2D radar detection objects; corresponding to the existence of radar detection objects that meet the overlap conditions, determining the obstructing objects and the obstructed objects among the radar detection objects that meet the overlap conditions; and determining the radar detection objects that do not meet the overlap conditions and the obstructing objects as the target radar detection objects whose obstruction degree meets the requirements.

2. The method according to claim 1, characterized in that, Determining whether there is a radar detector that satisfies the overlap condition among the at least two 2D radar detectors includes: identifying a first 2D radar detector and a second 2D radar detector with an overlapping region among the at least two 2D radar detectors; and determining the first 2D radar detector as a radar detector that satisfies the overlap condition if the area ratio of the overlapping region to the area of ​​the first 2D radar detector is greater than an area threshold.

3. The method according to claim 2, characterized in that, The step of identifying the obstructing and obstructed objects among radar detection objects that satisfy the overlap condition includes: determining the first depth of field and the second depth of field of the first 2D radar detection object and the second 2D radar detection object; and determining the first 2D radar detection object as the obstructing object and the second 2D radar detection object as the obstructed object when the first depth of field is less than the second depth of field.

4. The method according to claim 1, characterized in that, The recall conditions also include at least one of the following: the radar-detected object belongs to a predetermined type; the depth of field of the radar-detected object is less than or equal to a depth threshold.

5. The method according to claim 4, characterized in that, The predetermined type includes at least one of people and vehicles.

6. The method according to claim 1, characterized in that, The step of acquiring an image frame of the target scene captured at a first moment and determining that there are overexposed areas in the image frame includes: determining that there are M overexposed areas in M ​​different color spaces, and that there are overlapping areas among the M overexposed areas, where M is an integer greater than 1; and taking the overlapping areas as the overexposed areas of the image frame.

7. A readable storage medium, characterized in that, The readable storage medium includes one or more programs that, when executed on an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 6.

8. A program product, characterized in that, When the program product is executed on an electronic device, it causes the electronic device to implement the method described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: At least one memory for storing one or more programs; At least one processor is configured to execute the one or more programs to cause the electronic device to perform the method of any one of claims 1 to 6.

Images