Method and device for verifying target detection result, controller and product

Abstract

The embodiment of the invention relates to a method and device for verifying a target detection result, a controller and a product. The method comprises the step of determining a target detection result by performing target detection on an image. The method further comprises the step of determining the semantic segmentation result and the uncertainty corresponding to the pixels contained in the image by performing semantic segmentation on the image. And verifying a target detection result of the image based on the semantic segmentation result and the uncertainty. In this way, the target detection result can be verified in combination with different detection modes. Therefore, on one hand, the limitation of a single detection mode can be avoided, and on the other hand, the accuracy of target detection can be improved, so that the accuracy of the finally output target detection result is ensured.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image processing, and in particular to methods, devices, controllers, and products for verifying target detection results. Background Art

[0002] With the rapid development of computing and artificial intelligence, autonomous driving technology has become a key research area in the automotive industry. Autonomous driving technology primarily relies on sensors such as cameras and radar to perceive the surrounding environment, operating and performing necessary functions without human intervention, thereby achieving autonomous control of the vehicle.

[0003] Object detection is a key technology in autonomous driving. Object detection aims to determine the location of objects (such as vehicles, pedestrians, and road infrastructure) in three-dimensional space based on sensor data. Accurately determining the location of objects in three-dimensional space is fundamental to autonomous vehicles' perception of their surroundings and is crucial for safe autonomous driving. Summary of the Invention

[0004] Embodiments of the present disclosure provide a method, apparatus, controller, and product for verifying target detection results.

[0005] According to a first aspect of the present disclosure, a method for verifying object detection results is provided. The method includes performing object detection on an image to determine the object detection result. The method also includes performing semantic segmentation on the image to determine the uncertainty corresponding to the semantic segmentation result and pixels contained in the image. The method also includes verifying the object detection result of the image based on the semantic segmentation result and the uncertainty.

[0006] According to a second aspect of the present disclosure, a device for verifying an object detection result is provided. The device includes an object detection unit configured to determine an object detection result by performing object detection on an image; a semantic segmentation unit configured to determine a semantic segmentation result and an uncertainty corresponding to pixels contained in the image by performing semantic segmentation on the image; and a verification unit configured to verify the object detection result of the image based on the semantic segmentation result and the uncertainty.

[0007] According to a third aspect of the present disclosure, a controller is provided, comprising at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the controller to implement the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a computer program product is provided. The computer program product stores computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the method according to the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to the first aspect of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0011] Figure 1 A schematic diagram illustrating an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented;

[0012] Figure 2 A schematic diagram of performing object detection and semantic segmentation on an image according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A flowchart of a method for verifying target detection results according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram showing target pixels contained in a detection frame according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic diagram illustrating determining the distance between a detection box and a semantic box according to some embodiments of the present disclosure is shown;

[0016] Figure 6 A schematic block diagram of an apparatus for verifying target detection results according to an embodiment of the present disclosure is illustrated; and

[0017] Figure 7 A schematic block diagram illustrating one example of an example device suitable for implementing embodiments of the present disclosure is illustrated.

[0018] In the various drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0019] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] Object detection is an image processing technology that automatically identifies and locates target objects in an image by analyzing and processing the image. The task of object detection is to find all targets (objects) of interest in the image and determine their categories and locations. Because images are easily affected by background and environmental factors, such as changes in lighting and partial occlusion of the target, object detection cannot be perfect. It is easy for object detection results to be inaccurate, such as recognition errors or missing some targets, such as mistakenly identifying an object as a pedestrian when it is not a pedestrian. It is understandable that since the accuracy of object detection directly affects the functional execution of the autonomous driving system, if the object detection results are inaccurate, it may lead to the incorrect execution of the autonomous driving system, and in serious cases, it may even cause an accident, and the personal safety of the user cannot be guaranteed.

[0022] At least to address the above and other potential problems, an embodiment of the present disclosure provides a method for verifying target detection results. The method includes determining the target detection results by performing target detection on an image. The method also includes determining the uncertainty corresponding to the semantic segmentation results and the pixels contained in the image by performing semantic segmentation on the image. The method also includes verifying the target detection results of the image based on the semantic segmentation results and the uncertainty. In this way, the target detection results can be verified in combination with different detection methods. In this way, on the one hand, the limitations of a single detection method can be avoided, and on the other hand, the accuracy of target detection can be improved to ensure the accuracy of the target detection results ultimately outputted.

[0023] It should be noted that the embodiments of the present disclosure can be applied to various application fields, such as autonomous driving, facial recognition, and industrial inspection. Different application fields require different images to be captured and the objects contained in them. For example, in the autonomous driving field, the captured images are images of the vehicle's surrounding environment.

[0024] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Figure 1 Output is a schematic diagram of an example environment in which the device and / or method according to the embodiments of the present disclosure can be implemented. Figure 1 As shown, in Figure 1 In example environment 100, an image 102 and an electronic device 104 for verifying object detection results in image 102 are shown. Image 102 may be an image of the surrounding environment of a vehicle captured by a capture device. Image 102 may include one or more objects, such as pedestrians, buildings, traffic signs, lane markings, and the like.

[0025] In some embodiments, the acquisition device may be a vehicle. The vehicle may be equipped with an image acquisition device, which may be a camera or a still camera. In other embodiments, the image 102 may also be acquired through other acquisition devices (e.g., roadside acquisition devices). According to some embodiments of the present disclosure, the electronic device 104 may be a user terminal, a mobile device, a computer, etc. It may also be a computing system, a single server, a distributed server, or a cloud-based server. It should be understood that the environment 100 is merely illustrative and does not limit the scope of the present disclosure. The environment 100 may include devices not in the Figure 1 More components are shown, and various components in environment 100 may also be implemented in different ways.

[0026] According to some embodiments of the present disclosure, electronic device 104 includes a target detection unit 106, a determination unit 108, and a verification unit 110. Target detection unit 106 can perform target detection on input image 102 to determine a target detection result corresponding to image 102. The target detection result can be output in the form of a detection frame containing the corresponding target. Different detection frames represent the category and location information of different targets. The target detection result can include multiple detection frames.

[0027] According to some embodiments of the present disclosure, the determination unit 108 may perform semantic segmentation on the input image 102 to determine a semantic segmentation result corresponding to the image 102. Based on the semantic segmentation result, the uncertainty corresponding to the pixels contained in the image 102 may be determined. The semantic segmentation result may include the categories to which the pixels contained in the image 102 belong, and multiple regions (e.g., "pedestrian region," "building region," "background region," etc.) obtained by dividing the image 102 according to the categories to which different pixels belong (e.g., pedestrians, buildings, sky, etc.).

[0028] In some embodiments, the uncertainty of a pixel includes epistemic uncertainty and epistemic uncertainty. Epistemic uncertainty is used to characterize the uncertainty existing in the semantic segmentation model. For example, epistemic uncertainty can be used to characterize the uncertainty of the semantic segmentation model in classifying each pixel contained in the image into a certain category. For example, the uncertainty that pixel a belongs to "pedestrian" is 50%, then the epistemic uncertainty of the pixel a is 50%. In some embodiments, epistemic uncertainty is generally an error that already exists in the data. For example, epistemic uncertainty can be determined based on the uncertainty corresponding to the input data. Generally, epistemic uncertainty is reflected in the output of the semantic segmentation model. In one example, the epistemic uncertainty can be the epistemic uncertainty corresponding to the pixels in the edge area of ​​the image.

[0029] According to some embodiments of the present disclosure, the result output by the object detection unit 106 and the result output by the determination unit 108 may be input to the verification unit 110. The verification unit 110 may verify the object detection result based on the semantic segmentation result and the uncertainty, for example, verifying whether the object detection result is accurate (verifying whether the object in the object detection result is accurate or whether the position of the detection box is accurate). It should be understood that if the object detection result is found to be inaccurate by verification, the object detection result may be filtered out.

[0030] In this way, the target detection results can be verified by combining different detection methods. This can avoid the limitations of a single detection method on the one hand, and improve the accuracy and precision of target detection on the other hand, to ensure the accuracy of the target detection results ultimately output, thereby providing a more accurate reference for the control of the autonomous driving system. Of course, the embodiments of the present disclosure can also be applied to other application scenarios.

[0031] Figure 2 FIG2 shows a schematic diagram of performing target detection and semantic segmentation on an image according to some embodiments of the present disclosure. Figure 2As shown, the target detection network 204 can perform target detection on the input image 202 to determine the target detection result 208. The target detection result 208 may include a detection frame for the target. The target detection network 204 may be a target detection network or a target detection model trained using labeled data. For example, the target detection network 204 may be a convolutional neural network or other neural network. In some embodiments, the target detection network 204 may include a feature extraction layer, a regression network layer, and a classification network layer. The feature extraction layer is used to extract features from the image 202 to determine the image features corresponding to the image 202. The regression network layer is used to predict the position of the detection frame in the image 202 based on the image features. The classification network layer is used to predict the category to which the detection frame belongs based on the image features.

[0032] In some embodiments, the object detection result 208 output by the object detection network 204 can be input to the verification module 216 so that the verification module 216 can verify the object detection result 208 based on other information. The semantic segmentation network 206 can perform semantic segmentation on the input image 202 to determine a semantic segmentation result 210. The uncertainty estimation network 212 is configured to determine the uncertainty 214 corresponding to the pixels in the image 202 based on the semantic segmentation result 210. The semantic segmentation network 206 can be a neural network or other machine learning algorithm model. If a neural network is used as the semantic segmentation network, a convolutional neural network or other type of neural network can be used. The semantic segmentation network 206 can input the semantic segmentation result 210 to the uncertainty estimation network 212, so that the uncertainty estimation network 212 can determine the uncertainty 214 corresponding to the pixels in the image 202 based on the semantic segmentation result 210. The semantic segmentation result 210 output by the semantic segmentation network 206 and the uncertainty 214 output by the uncertainty estimation network 212 can be input to the verification module 216. The verification module 216 can verify whether the object detection result 208 is accurate based on three types of input information (e.g., the object detection result 208, the semantic segmentation result 210, and the uncertainty 214). For example, the object detection result 208 may include a detection box for the object. Based on the position information of the detection box in the object detection result 208, the semantic segmentation result 210 and the uncertainty 214 at the same location can be determined. Based on the object detection result 208, the semantic segmentation result 210, and the uncertainty 214 at the same location, the object detection result 208 at that location is verified to be accurate.

[0033] In some embodiments of the present disclosure, if the majority of target pixels contained in the detection frame belong to the same category as the target, and the uncertainty corresponding to most of the target pixels is low, it can be determined that the detection result is accurate. If the majority of target pixels contained in the detection frame belong to categories different from the target, and the uncertainty corresponding to most of the target pixels is high, it can be determined that the detection result is inaccurate. In addition, the subsequent processing of inaccurate target detection results can be different according to different settings of the autonomous driving system. For example, if the autonomous driving system tends to conservative safety settings, the inaccurate target detection results are retained; if the autonomous driving system tends to aggressive safety settings, the inaccurate target detection results are removed.

[0034] Figure 3 FIG. 3 is a flow chart of a method 300 for verifying target detection results according to an embodiment of the present disclosure. It should be noted that the method 300 according to an embodiment of the present disclosure may be, for example, Figure 1 The electronic device 104 shown in FIG.

[0035] In box 302, target detection is performed on the image to determine the target detection result. The image may be an image of the surrounding environment obtained by capturing the image through the vehicle. The vehicle may be equipped with an image capture device (such as a camera). In addition, other capture devices may be used to capture images of the surrounding environment of the vehicle. It should be noted that the image may include multiple images of the surrounding environment from different angles acquired at the same time. In addition, the image capture device or other capture device may capture an image or a video stream of the surrounding environment. When the video stream is captured, after obtaining the video stream, the video stream may be decoded to obtain a plurality of frames of images, and then the image may be obtained from the plurality of frames of images.

[0036] In some embodiments, target detection refers to detecting the location (Location) of a target in an image and determining the category (Classification) of the target. The target detection result may include a detection box or bounding box (Bounding Box) for a target and the category information of the target (for example, the category information is "pedestrian"). The category information can be in various forms, such as numerical values, colors, etc. For example, 1 can represent a pedestrian, 2 can represent a vehicle, and 3 can represent a building. In some embodiments, the location information of the detection box can be determined by the coordinates of the upper left corner and the coordinates of the lower right corner of the detection box (or by the coordinates of the upper right corner and the coordinates of the lower left corner of the detection box). The detection box is the smallest rectangular box that can just contain the target. In some embodiments, a target detection network can be used to perform target detection on an image. The target detection network can be a neural network or other machine learning algorithm model, and the neural network can include a convolutional neural network or other types of neural networks.

[0037] In box 304, semantic segmentation is performed on the image to determine the uncertainty corresponding to the semantic segmentation result and the pixels contained in the image. Herein, semantic segmentation of the image refers to classifying each pixel in the image according to its semantic category and determining the category to which each pixel belongs, thereby achieving pixel-level semantic understanding. In some embodiments, semantic segmentation can be performed on the image by a semantic segmentation model or a semantic segmentation network to determine the semantic segmentation result. The semantic segmentation result can be the category information corresponding to the pixels contained in the image. The form used to represent the category information can be various pre-set forms (such as characters, colors, numerical values, etc.), and these representation forms can distinguish different category information. For example, a preset color (for example, red) can be used to represent the category of "pedestrians".

[0038] In some embodiments, due to the discrepancy between the training data and the actual data to be detected, the generated labels inevitably contain incorrect predictions. Therefore, each pixel in the image will have an uncertainty. This uncertainty can be the confidence level in classifying the pixel, or it can be the uncertainty corresponding to pixels in the edge region of the image. The uncertainty corresponding to pixels in the edge region of the image is called accidental uncertainty. This uncertainty may be primarily due to the object in the image being far away from the acquisition device, resulting in blurry images at the edges.

[0039] In some embodiments, a higher uncertainty corresponding to a pixel indicates a higher confidence level that the pixel is correctly classified into a certain category. Conversely, a lower uncertainty corresponding to a pixel may indicate a lower confidence level that the pixel is correctly classified into a certain category. In one example, the probability that a pixel belongs to the semantic feature "pedestrian" is 10%, and the uncertainty of the pixel is 90%. The uncertainty of the pixel can then be expressed in the form of (10%).

[0040] In block 306, the object detection results for the image are verified based on the semantic segmentation results and the uncertainty. Verifying the object detection results involves verifying the accuracy of the object detection results, for example, verifying the accuracy of the position or category of a detection box. In some embodiments, because the image targeted by object detection and the image targeted by semantic segmentation are the same image, the semantic segmentation results and the object detection results contain the same pixels and pixel positions. Based on this, the semantic segmentation results and uncertainties at the same locations can be determined based on the position information of the detection boxes. The object detection results are verified based on the semantic segmentation results and uncertainties at the same locations.

[0041] This approach allows for the integration of multiple detection methods into images of the vehicle's surroundings, avoiding the limitations of a single detection method. This improves detection precision and accuracy, and reduces false positives. Furthermore, accurately identifying objects within an image can provide a reference for controlling the autonomous driving system, improving its accuracy.

[0042] In some embodiments, the vehicle may also include one or more other sensors such as a Global Navigation Satellite System (GNSS) and an Inertial Measurement Unit (IMU) to record information such as the time and location of image acquisition. Specifically, the image acquisition device is mainly used to capture images of pedestrians, vehicles, roads, greenery and other targets in the target environment. Images can include any format such as BMP, JPEG, PNG, SVG, etc. GNSS can be used to record the coordinates of the currently captured image and point cloud data. IMU is mainly used to record the angle and acceleration information of the captured vehicle. All of the above information can be used as auxiliary information to assist in the target detection of the image, thereby improving the precision and accuracy of target detection.

[0043] In some embodiments, the result of semantic segmentation includes semantic information of each pixel in the image. The semantic information may include the semantic features corresponding to the pixel and the category to which the pixel belongs. The categories include, for example, pedestrians, vehicles, roads, trees, buildings and the like. The semantic features of a pixel may be any features that can characterize the category of the pixel. Since the processing objects of target detection and semantic segmentation are the same image. The position information of the pixels contained in the image is fixed and does not change with the change of the detection method. Therefore, the target pixels contained in the detection frame can be determined based on the position information of the detection frame for a certain target (for example, {x, y}). The category to which each target pixel belongs can be determined based on the semantic segmentation result. Furthermore, the uncertainty of each target pixel can be determined based on the uncertainty of the pixels contained in the image.

[0044] In some embodiments, since the target detection result also includes the target's category information, the target detection result can be verified based on the comparison between the target pixel category information and the target category information. If the category of most target pixels in the detection frame is different from the category of the target, the target detection result is inaccurate. If the category of most target pixels in the detection frame is the same as the category of the target, the target detection result is accurate.

[0045] In some embodiments, in order to more accurately verify the target detection results and measure the accuracy of the target detection results, the target detection results can be verified based on the number of target pixels in the detection frame that share the same category as the target and the total number of target pixels contained in the detection frame. For example, the target detection results can be verified based on the difference or ratio between the number and the total number. If the difference or ratio is greater than a preset threshold, the target detection result is determined to be accurate. It will be understood that the difference between the difference and the preset threshold can be used to measure the accuracy of the target detection results.

[0046] Figure 4 Schematic diagram of target pixels contained in a detection frame according to some embodiments of the present disclosure is shown. Figure 4 As shown, the target detection result 402 includes a detection frame for target A. The rectangular frame in the semantic segmentation result 404 corresponds to the position of the detection frame. The total number of target pixels 406 contained in the rectangular frame is k. Among them, the number of target pixels belonging to target A is m, the number of target pixels that do not belong to target A is n, and n+m=k. If m / k is greater than the preset threshold, it indicates that the target detection result is accurate. In this way, the semantic segmentation result of the image can be combined to assist in verifying the target detection result of the image, thereby improving the accuracy of target detection.

[0047] In some embodiments of the present disclosure, in order to further improve the accuracy of verifying the target detection results, the target detection results can also be verified based on the uncertainty of the target pixels contained in the detection frame. It can be understood that the target pixel can also be a target pixel whose category in the detection frame corresponds to the category of the target. In some embodiments, the target detection result can be verified based on the comparison result of the uncertainty of the target pixel and a predetermined confidence threshold. In one example, if the statistical value of the uncertainty of the target pixels contained in the detection frame is greater than the predetermined uncertainty threshold, it is determined that the detection of the detection frame is not accurate, that is, the target detection result is not accurate. The statistical value can be the sum or average of the uncertainties corresponding to all target pixels.

[0048] Table 1 below shows an uncertainty table corresponding to multiple target pixels according to some embodiments of the present disclosure.

[0049] Table 1 Uncertainty table corresponding to target pixels

[0050]

[0051]

[0052] Among them, the pixel identifier is used to uniquely identify a certain pixel point. When the target pixels contained in the detection frame are 00, 01, 02, and 03, and the corresponding uncertainties are 7, 5, 3, and 9 respectively, the statistical value can be 6. If the predetermined uncertainty threshold is 10, the detection frame is accurate (the target detection result is accurate). In other embodiments, in order to determine a more accurate verification result, it is also possible to determine whether the detection frame is accurate or inaccurate or how accurate it is based on the ratio of the statistical value to the area of ​​the detection frame. Among them, the area of ​​the detection frame can be determined based on the number of pixels contained in the detection frame.

[0053] In this way, the target detection results can be verified more accurately based on the uncertainty corresponding to the pixels contained in the detection frame, thereby further optimizing the accuracy of target detection.

[0054] In some embodiments, the image can be divided into regions according to the category to which each pixel belongs, and different regions can be obtained. For example, the category to which regions 1 and 3 belong is a building, the category to which regions 2 and 5 belong is a vehicle, and the category to which image blocks 4 and 6 belong is a pedestrian. The boundary line of the region can constitute a semantic frame of the region. On this basis, the target detection result can be verified based on the detection frame and the corresponding semantic frame. In some embodiments, the region that meets the requirements is determined in the semantic segmentation result based on the position information of the detection frame. Meeting the requirements means that the region is within the range corresponding to the detection frame. Among them, scaling the detection frame can determine the range corresponding to the detection frame. After determining the region that meets the requirements, the corresponding semantic frame can be determined based on the bounding box formed by the target pixels of the target (the target contained in the detection frame) in the region.

[0055] In some embodiments, the accuracy of the target detection result can be verified based on the similarity between the semantic box and the detection box. The similarity between the semantic box and the detection box can be calculated using a similarity calculation method. Similarity calculation methods may include, for example, cosine similarity, Euclidean distance, Pearson correlation coefficient, and Manhattan distance. In other embodiments, the accuracy of the target detection result can also be verified based on the distance between the semantic box and the detection box. The distance between the semantic box and the detection box can be determined based on the distance between the geometric center of the semantic box and the geometric center of the detection box.

[0056] Figure 5 FIG. 5 shows a schematic diagram of determining the distance between a detection box and a semantic box according to some embodiments of the present disclosure. Figure 5As shown, based on the position of the detection frame 502 and the corresponding target, the semantic frame 504 can be determined. The distance between the detection frame 502 and the semantic frame 504 can be determined based on the distance between the center point A of the detection frame 502 and the center point B of the semantic frame 504. The distance d can be determined based on the distance formula (1):

[0057]

[0058] Where (x1, y1) is the position coordinate of the center point A, and (x2, y2) is the position coordinate of the center point B.

[0059] In order to further improve the accuracy of verifying the target detection results, the target detection results can be double-verified based on the semantic segmentation results and the uncertainty to ensure that the verification results are accurate. For example, the target detection results can be verified for the first time based on the semantic segmentation results to determine the first verification result. The first verification result can be a specific verification result, for example, the categories of most target pixels contained in the detection box for the target in the target detection result are the target (or the categories of most target pixels are different from the target). Of course, the first verification result can also be a simple verification conclusion (accurate or inaccurate). The target detection results are verified for the second time based on the uncertainty to determine the second verification result. The second verification result can be a specific verification result, for example, the uncertainty of the target pixels contained in the detection box for the target in the target detection result is less than a predetermined confidence threshold (or the uncertainty of the target pixels is greater than a predetermined confidence threshold). Of course, the second verification result can also be a simple verification conclusion (accurate or inaccurate).

[0060] If both verification results show that the target detection result is accurate, then the target detection result is determined to be accurate. If one of the two verification results is accurate and the other is inaccurate, the final verification result can be determined based on the specific situation, and the subsequent processing method for the target detection result can be determined. For example, whether the final target detection result is accurate can be determined based on the priority of the first verification result and the second verification result. For example, when the first verification result is accurate and the second verification result is inaccurate, and the priority of the first verification result is greater than the priority of the second verification result, it can be determined that the target detection result is accurate. In other embodiments, whether the target detection result is accurate can be determined based on the function setting of the autonomous driving. If the function setting of the autonomous driving tends to be safe, then the target detection result is determined to be inaccurate.

[0061] In some embodiments, in order to make the control of autonomous driving more accurate, the credibility of the target can be determined based on the verification results. The credibility can be used to characterize the possibility of the target existing at the location or the possibility of the target appearing at the location. In some embodiments, the credibility can be determined based on the verification results. The verification results may include the verification conditions and the number of conditions that are met. In some embodiments, the verification conditions may include whether the number of pixels belonging to the target contained in the detection frame is greater than a predetermined number, whether the uncertainty of each pixel contained in the detection frame is greater than a predetermined uncertainty, whether the size or dimension of the detection frame conforms to the empirical size, and whether the similarity between the semantic frame and the detection frame is greater than a predetermined similarity. In some embodiments, if all of the above verification conditions are met, the credibility of the target is high. If all of the above verification conditions are not met, the credibility of the target is reduced accordingly. If the number of conditions that meet the above verification conditions is greater than a set value, the credibility of the target is increased accordingly.

[0062] In this way, object detection results can be double-verified based on semantic segmentation results and uncertainty. On the one hand, this can determine more accurate verification results. On the other hand, it can more fully consider the verification requirements of different application scenarios, thereby determining verification results that meet the requirements under different verification requirements.

[0063] It should be understood that if the final verification result is inaccurate, the detection box for the target in the target detection result can be deleted to prevent the detection box from affecting subsequent processing. In some embodiments, in order to improve the accuracy of the target detection results outputted subsequently, the target detection network can be adjusted according to the verification result so that the target detection results outputted by the adjusted target detection network are more accurate.

[0064] In some embodiments, verifying the target detection result based on the semantic segmentation result can be verifying the target detection result based on the ratio of the size of the detection box to the size of the pixels belonging to the target. For example, in one example, when the size of the detection box is 40 pixels * 60 pixels, the number of pixels belonging to the target in the detection box is 1800, and the number of pixels belonging to the target not in the detection box is 200, the ratio can be 1800 / (40*60). It can be understood that the larger the ratio, the more accurate the detection box is, that is, the target detection result is accurate; and when the ratio is smaller, it proves that the target detection result is not accurate. According to the size of the ratio, the inaccuracy of the target detection result can also be measured.

[0065] In some embodiments, in order to clearly and intuitively determine the uncertainty corresponding to different pixels, a heat map can be generated based on the semantic segmentation results. The heat map is used to characterize the distribution of uncertainty. Each pixel in the heat map is marked with a corresponding uncertainty. In some embodiments, a heat map generation model can be used to determine the corresponding heat map based on the input semantic segmentation results. In some embodiments, the uncertainty corresponding to different pixels in the heat map can be represented in the form of color, grayscale, brightness, etc. For example, the uncertainty corresponding to the pixel can be converted into the corresponding brightness and displayed in the heat map. Pixels with lower uncertainty can be converted into pixels with lower brightness, while pixels with higher uncertainty can be converted into pixels with higher brightness.

[0066] For another example, the uncertainty corresponding to the pixel can be converted into the corresponding color and displayed in the heat map. Among them, the pixels with higher uncertainty can be converted into green pixels, and the pixels with lower uncertainty can be converted into blue pixels. Based on this, the color distribution in the heat map can be used to characterize the distribution of uncertainty. Therefore, the target detection result can be verified according to the color distribution of the heat map. For example, the corresponding target area can be determined in the heat map according to the position information and size of the detection box. According to the color distribution of the target area, the target detection result can be verified. For example, if the color of the target area is mostly blue, it can be determined that the target detection result is accurate.

[0067] This approach provides a more direct way to verify target detection results. The uncertainty corresponding to different pixels can be intuitively determined in the form of a heat map.

[0068] In actual applications, each object or subject has its own regular shape or size. For example, the length of a small car is generally between 1.5m and 3m. If the length of the detection frame for a small car included in the target detection result is 50cm, which does not conform to the normal size of a small car, then the detection result is not accurate. In view of this, in some embodiments of the present disclosure, the target detection result can be verified based on the empirical size of the target to determine whether the target detection result is accurate. For example, the accuracy of the target detection result can be determined based on whether the size information of the detection frame for the target is consistent with the empirical size. The size information can be the length, width, height, and aspect ratio of the detection frame. For example, when the height and width of the vehicle's detection frame {h, w} conform to the empirical size of the vehicle, it can be determined that the obtained detection frame is accurate. In some embodiments, if the target detection result is verified to be accurate, the target detection result can be retained; and if the target detection result is verified to be inaccurate, the target detection result can be filtered (for example, deleted).

[0069] In some embodiments, the accuracy of the detection frame can also be determined based on the categories of the pixels surrounding the detection frame. For example, if the object contained in the detection frame is a "vehicle," but most of the pixels in the area surrounding the detection frame belong to the "sky" category, the detection frame can be determined to be inaccurate, and therefore, the probability of a vehicle appearing in the sky is very, very low.

[0070] In this way, some target detection results that are obviously inconsistent with common sense can be eliminated, and more accurate target detection results can be obtained, thereby improving the accuracy of the target detection results.

[0071] Figure 6 600 is a schematic block diagram of an apparatus for verifying target detection results according to an embodiment of the present disclosure. Figure 6 As shown, the device 600 includes a target detection unit 602, which is configured to determine the target detection result by performing target detection on the image; a semantic segmentation unit 604, which is configured to determine the semantic segmentation result and uncertainty by performing semantic segmentation on the image; and a verification unit 606, which is configured to verify the target detection result of the image based on the semantic segmentation result and the uncertainty.

[0072] In some embodiments, the target detection result includes a detection box for the target, the semantic segmentation result includes the semantic information of the pixels contained in the image, and the verification unit 606 is further configured to: determine the semantic information of the target pixels contained in the detection box and the uncertainty of the target pixels based on the semantic information of the pixels contained in the image; and verify the target detection result of the image based on the semantic information of the target pixels and the uncertainty of the target pixels.

[0073] In some embodiments, the verification unit 606 is further configured to: determine the number of target pixels belonging to the target in the detection frame and the total number of target pixels contained in the detection frame; and verify the target detection result of the image based on the number and the total number.

[0074] In some embodiments, the verification unit 606 is further configured to: compare the statistical value of the uncertainty of the target pixel with a predetermined uncertainty threshold; and verify the target detection result of the image based on the comparison result.

[0075] In some embodiments, the target detection result includes a detection box for the target, and the verification unit 706 is further configured to: determine the semantic box corresponding to the target in the semantic segmentation result based on the position information of the detection box and the target; and verify the target detection result of the image based on the detection box and the semantic box.

[0076] In some embodiments, the verification unit 606 is further configured to: verify the target detection result of the image through the semantic segmentation result to determine the first verification result; verify the target detection result of the image through the uncertainty to determine the second verification result; and verify whether the target detection result of the image is accurate based on the first verification result and the second verification result.

[0077] In some embodiments, the verification unit 606 is further configured to: in response to the first verification result being accurate and the second verification result being accurate, determine that the object detection result of the image is accurate.

[0078] In some embodiments, the semantic segmentation unit 604 is further configured to: determine a heat map corresponding to the image by performing semantic segmentation on the image; and determine the uncertainty of pixels contained in the image based on the heat map.

[0079] In some embodiments, the verification unit 606 is further configured to: determine the target area corresponding to the area in the heat map based on the area corresponding to the detection box for the target contained in the target detection result; and verify the target detection result of the image based on the color distribution of the target area.

[0080] In some embodiments, the apparatus 600 further includes a second verification unit configured to: determine the size of a detection frame for the target based on the target detection result; and verify the target detection result of the image based on the empirical size of the target and the size of the detection frame.

[0081] In some embodiments, the device 600 also includes a detection frame deletion unit, which is configured to: determine a corresponding verification result by verifying the target detection result of the image, where the verification result includes one of accurate and inaccurate; and when the verification result is inaccurate, delete the detection frame included in the target detection result.

[0082] In some embodiments, the uncertainty includes epistemic uncertainty and aleatory uncertainty.

[0083] The device according to the embodiments of the present disclosure can combine different detection methods to verify the target detection results. This can avoid the limitations of a single detection method on the one hand, and improve the accuracy and precision of target detection on the other hand, to ensure the accuracy of the target detection results ultimately output.

[0084] Figure 7Shown is a schematic block diagram of an example device 700 suitable for implementing an embodiment of the present disclosure. The controller above can be implemented using device 700. As shown, device 700 includes a processor 701, which can be loaded into a computer program instruction in a random access memory (RAM) 703 according to the computer program instruction stored in a read-only memory (ROM) 702, to perform various appropriate actions and processes. In RAM 703, various programs and data required for the operation of device 700 can also be stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0085] The various processes and procedures described above, such as method 300, may be executed by processor 701. For example, in some embodiments, method 300 may be implemented as a computer software program tangibly embodied on a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 700 via ROM 702. When the computer program is loaded into RAM 703 and executed by processor 701, one or more actions of method 300 described above may be performed.

[0086] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0087] Computer-readable storage media can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage media used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by a fiber optic cable), or an electrical signal transmitted by a wire.

[0088] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0089] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0090] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0091] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0092] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0093] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0094] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0095] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method (300) for verifying target detection results, comprising: Determining a target detection result by performing target detection on the image (302); Determining a semantic segmentation result and an uncertainty corresponding to pixels contained in the image by performing semantic segmentation on the image (304); as well as Based on the semantic segmentation result and the uncertainty, the object detection result of the image is verified (306).

2. The method (300) according to claim 1, wherein the object detection result includes a detection box for the object, the semantic segmentation result includes semantic information of pixels contained in the image, and based on the semantic segmentation result and the uncertainty, verifying (306) the object detection result of the image comprises: Determining semantic information of a target pixel contained in the detection frame and uncertainty of the target pixel based on semantic information of pixels contained in the image; as well as The target detection result of the image is verified based on the semantic information of the target pixel and the uncertainty of the target pixel.

3. The method (300) according to claim 2, wherein verifying (306) the object detection result of the image based on the semantic information of the target pixel and the uncertainty of the target pixel comprises: Determine the number of target pixels in the detection frame that belong to the target and the total number of target pixels contained in the detection frame; as well as Based on the number and the total number, an object detection result of the image is verified.

4. The method (300) of claim 2, wherein verifying (306) the object detection result of the image based on the semantic information of the target pixel and the uncertainty of the target pixel comprises: comparing the statistical value of the uncertainty of the target pixel with a predetermined uncertainty threshold; as well as The object detection result of the image is verified based on the comparison.

5. The method (300) according to claim 1, wherein the object detection result includes a detection box for the object, and based on the semantic segmentation result and the uncertainty, verifying (306) the object detection result of the image comprises: Determining a semantic frame corresponding to the target in the semantic segmentation result based on the position information of the detection frame and the target; as well as An object detection result of the image is verified based on the detection box and the semantic box.

6. The method (300) according to claim 1, wherein verifying (306) the object detection result of the image based on the result of the semantic segmentation and the uncertainty comprises: Verifying the object detection result of the image by using the semantic segmentation result to determine a first verification result; Verifying the target detection result of the image by the uncertainty to determine a second verification result; as well as Based on the first verification result and the second verification result, verify whether the object detection result of the image is accurate.

7. The method (300) according to claim 6, wherein verifying (306) whether the object detection result of the image is accurate based on the first verification result and the second verification result comprises: In response to the first verification result being accurate and the second verification result being accurate, it is determined that the object detection result of the image is accurate.

8. The method (300) according to claim 1, wherein determining a semantic segmentation result and an uncertainty by performing semantic segmentation (304) on the image comprises: Determine the semantic segmentation result by performing semantic segmentation on the image; Determining a heat map corresponding to the image based on the semantic segmentation result; as well as Based on the heat map, the uncertainty corresponding to the pixels included in the image is determined.

9. The method (300) of claim 8, wherein verifying (306) the object detection result of the image based on the semantic segmentation result and the uncertainty comprises: Based on the area corresponding to the detection box for the target included in the target detection result, determining a target area corresponding to the area in the heat map; as well as The target detection result of the image is verified based on the color distribution of the target area.

10. The method (300) of claim 1, further comprising: Determining a size of a detection frame for the target based on a result of the target detection; as well as The object detection result of the image is verified based on the empirical size of the object and the size of the detection frame.

11. The method (300) of claim 1 , further comprising: Determine a corresponding verification result by verifying the object detection result of the image, wherein the verification result includes whether it is accurate or inaccurate; as well as If the verification result is inaccurate, the detection box included in the target detection result is deleted.

12. The method (300) of claim 1, wherein the uncertainty comprises epistemic uncertainty and random uncertainty.

13. A device (600) for verifying a target detection result, comprising: The target detection unit (602) is configured to determine a target detection result by performing target detection on the image; A semantic segmentation unit (604) is configured to determine a semantic segmentation result and an uncertainty corresponding to pixels contained in the image by performing semantic segmentation on the image; as well as A verification unit (606) is configured to verify the object detection result of the image based on the semantic segmentation result and the uncertainty.

14. A controller comprising: at least one processor; as well as A memory is coupled to the at least one processor and has instructions stored thereon, which, when executed by the at least one processor, cause the controller to perform the method according to any one of claims 1-12.

15. A computer program product comprising computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the method according to any one of claims 1 to 12.

Images