Target detection method and device
By fusing the confidence of the target in a single image and the circum-view image, the problem of low object detection confidence in the circum-view image is solved, the accuracy of object detection is improved, and the credibility of the detection results is ensured.
Patent Information
- Application Number
- CN201910498217.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-06-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-06-10
AI Technical Summary
When object detection is performed based on circum viewing images, the confidence of the target is too low, resulting in inaccurate detection results and prone to object detection errors.
By fusing the confidence of the target detected from a single image and the confidence of the target detected from the circumferential image, the fusion confidence is obtained to ensure the credibility of the detected target, thereby improving the accuracy of the target detection. sex. The specific method includes acquiring the fusion confidence based on the first confidence level and the second confidence level, and outputting the target detection result based on the fusion confidence level, and determining the target positioning information.
Improve the accuracy of target detection, ensure the credibility of detected targets, and reduce the situation of false detection and missed detection.
Smart Images

Figure CN112069862B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to automobile intelligent assisted driving technology, and in particular to a target detection method and device. Background Art
[0002] With the development of society, smart cars are gradually entering people's daily lives. Sensors play a very important role in the assisted driving and automatic driving of smart cars. Various sensors installed on the car, such as millimeter wave radar, laser radar, camera, ultrasonic radar, etc., can sense the surrounding environment at any time during the driving process of the car, collect data, identify and track moving objects, and identify static scenes such as lane lines and signs, and plan routes in combination with navigation and map data. Sensors can detect possible dangers in advance and help drivers in time or even take necessary avoidance measures autonomously, effectively increasing the safety and comfort of car driving.
[0003] In the related technology, multiple cameras installed at different positions on the vehicle body are used to take pictures respectively, so as to obtain images of the front, back, left and right angles of the vehicle, and then these images are stitched together to obtain a panoramic surround image, which the driver can watch on the large screen of the vehicle's central control to understand the surrounding environment of the vehicle. By performing target detection on the surround image through a trained neural network, the position of the target (such as a parking space, a person, an obstacle, a lane line, etc.) is determined, and the recall rate and accuracy of the detection result are high, thereby realizing intelligent assisted driving of the vehicle.
[0004] However, when performing target detection based on surround view images, the confidence level of the target in the detection results is too low, and it is impossible to determine whether it is a real target, which easily leads to target detection errors. Summary of the invention
[0005] The present application provides a target detection method and device to ensure the credibility of the detected target by fusing the confidence of the target detected from a single image with the confidence of the target detected from the surround image, thereby improving the accuracy of target detection.
[0006] In a first aspect, the present application provides a target detection method, comprising:
[0007] When the first target and the second target are the same target, a fusion confidence is obtained according to the first confidence and the second confidence; wherein the first confidence is the credibility of the first target in the first image, the second confidence is the credibility of the second target in the second image, the first image is an image from a first camera device, and the second image is a surround view image obtained based on images from multiple cameras, and the multiple cameras include the first camera device; the target detection result is output according to the fusion confidence.
[0008] The present application ensures the credibility of the detected target by fusing the confidence of the target detected from a single fisheye image with the confidence of the target detected from the surround view image, thereby improving the accuracy of target detection.
[0009] In one possible implementation, obtaining a fusion confidence based on the first confidence and the second confidence includes: when the first confidence is greater than a first threshold and the second confidence is greater than a second threshold, determining the fusion confidence according to a preset weight value; and / or, when the first confidence is less than or equal to the first threshold and the second confidence is greater than the second threshold, determining the fusion confidence to be the second confidence.
[0010] In a possible implementation, outputting the target detection result according to the fusion confidence includes: determining that the fusion confidence is greater than a third threshold; and outputting positioning information of the second target detected in the second image.
[0011] The present application improves the accuracy of target detection by inputting the positioning information of the second target detected in the second image based on the fusion confidence.
[0012] In a possible implementation, obtaining a fusion confidence based on the first confidence and the second confidence includes: when the first confidence is greater than the first threshold and the second confidence is less than or equal to the second threshold, determining that the fusion confidence is the first confidence.
[0013] In a possible implementation, outputting the target detection result according to the fusion confidence includes: determining that the fusion confidence is greater than a third threshold; and outputting positioning information of the first target detected in the first image.
[0014] The present application improves the accuracy of target detection by integrating the positioning information of the first target detected in the first image with the confidence input.
[0015] In a possible implementation, it also includes: determining whether the first target and the second target are the same target based on the image overlap of the first target and the second target; when the image overlap of the first target and the second target is greater than a fourth threshold, determining that the first target and the second target are the same target.
[0016] The present application determines whether the first target and the second target are the same target based on the image overlap degree of the first target and the second target, thereby improving the accuracy of target judgment.
[0017] In a possible implementation manner, the method further includes: acquiring the first image from the first camera device; and stitching the images from the multiple camera devices to obtain the second image.
[0018] In a second aspect, the present application provides a target detection device, characterized in that it includes:
[0019] A processing module is configured to obtain a fused confidence based on a first confidence and a second confidence when the first target and the second target are the same target; wherein the first confidence is the credibility of the first target in the first image, the second confidence is the credibility of the second target in the second image, the first image is an image from a first camera device, and the second image is a surround view image obtained based on images from multiple cameras, and the multiple cameras include the first camera device; an interface module is configured to output a target detection result based on the fused confidence.
[0020] In one possible implementation, the processing module is configured to determine the fusion confidence according to a preset weight value when the first confidence is greater than a first threshold and the second confidence is greater than a second threshold; and / or, when the first confidence is less than or equal to the first threshold and the second confidence is greater than the second threshold, determine that the fusion confidence is the second confidence.
[0021] In a possible implementation, the processing module is configured to determine that the fusion confidence is greater than a third threshold; and the interface module is configured to output positioning information of the second target detected in the second image.
[0022] In a possible implementation, the processing module is configured to determine that the fusion confidence is the first confidence when the first confidence is greater than the first threshold and the second confidence is less than or equal to the second threshold.
[0023] In a possible implementation, the processing module is configured to determine that the fusion confidence is greater than a third threshold; and the interface module is configured to output positioning information of the first target detected in the first image.
[0024] In one possible implementation, the processing module is configured to determine whether the first target and the second target are the same target based on the image overlap of the first target and the second target; when the image overlap of the first target and the second target is greater than a fourth threshold, it is determined that the first target and the second target are the same target.
[0025] In a possible implementation, the processing module is configured to acquire the first image from the first camera device; and to stitch images from the multiple camera devices to obtain the second image.
[0026] In a third aspect, the present application provides a device, including:
[0027] one or more processors;
[0028] A memory for storing one or more programs;
[0029] When the one or more programs are executed by the one or more processors, the device implements the method as described in any one of the first aspects above.
[0030] In a fourth aspect, the present application provides a computer-readable storage medium, comprising a computer program, wherein when the computer program is executed on a computer, the computer executes any one of the methods described in the first aspect.
[0031] In a fifth aspect, the present application provides a computer program, which, when executed by a computer, implements any one of the methods in the first aspect.
[0032] In a sixth aspect, the present application provides a device comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to implement a method as described in any one of the above-mentioned first aspects. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A schematic diagram of a target detection system applicable to the target detection method of this application;
[0034] Figure 2 This is a schematic diagram of the sight line direction of the camera device of this application;
[0035] Figure 3 This is a flow chart of an embodiment of the target detection method of the present application;
[0036] Figure 4A and 4B A schematic diagram showing a fisheye image and a corrected image;
[0037] Figure 5 This is a schematic diagram of the detection result of the first image of this application;
[0038] Figure 6 This is a schematic diagram of the detection result of the second image of this application;
[0039] Figure 7 It is another schematic diagram of the detection result of the first image of the present application;
[0040] Figure 8 This is a schematic diagram of the detection results of the first image and the second image of the present application;
[0041] Fig. 9 This is a schematic diagram of the overlapping area of two BBs in this application;
[0042] Fig.10 This is a schematic diagram of the combined area of the two BBs for this application;
[0043] Fig.11 A flowchart for fusing the confidence of the first image and the second image for the present application;
[0044] Fig.12 This is a schematic diagram of the structure of an embodiment of a target detection device of the present application;
[0045] Fig.13 A schematic block diagram of a target detection device provided in this application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] The terms "first", "second", etc. in the specification embodiments, claims, and drawings of the present application are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.
[0048] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0049] The following are some descriptions of terms involved in this application, and the meanings of the characteristics of the well-known terms:
[0050] Fisheye image: Fisheye image is a three-dimensional picture, which is less affected by the environment and can effectively capture space scenes and ground scenes. Fisheye image is an image taken by a fisheye lens. Fisheye lens is a lens with a focal length less than or equal to 16mm and a viewing angle close to or equal to 180°. It is an extreme wide-angle lens. In order to achieve the maximum photographic viewing angle, the front lens of the fisheye lens has a very short diameter and protrudes toward the front of the lens in a parabolic shape, which is quite similar to the eyes of fish, hence the name "fisheye lens".
[0051] Stitching: Images from multiple cameras are stitched together by using the image fusion and stitching algorithm based on the corresponding image relationship of the overlapping areas in the images from multiple cameras.
[0052] Confidence: When detecting a target, it describes the credibility of the detection result, which indicates the credibility of the detected target. For example, the value range is between 0 and 1.
[0053] Figure 1 A schematic diagram of a target detection system applicable to the target detection method of this application is shown in FIG. Figure 1 As shown, the target detection system may include multiple camera devices and a target detection device, wherein the multiple camera devices may be distributed and installed around the vehicle body, and the camera coverage areas of two adjacent camera devices may partially overlap, so as to facilitate the stitching of the surround view images described below. The target detection device may be independently set, or may be integrated in the control device, or may be implemented by software or a combination of software and hardware, and there is no specific limitation on this. Optionally, the aforementioned target detection system may be applied to an Advanced Driver Assistance System (ADAS).
[0054] Exemplarily, the above-mentioned multiple cameras may adopt a four-way fisheye camera installed on the vehicle body, wherein the front fisheye camera may be installed in the upper middle part of the vehicle logo, and the rear fisheye camera may be installed in the upper middle part of the vehicle license plate, and it is necessary to ensure that the front and rear cameras are located on the central axis of the vehicle (the error may be within plus or minus 5 mm). The left fisheye camera and the right fisheye camera may be installed under the left and right rearview mirrors of the vehicle, respectively. It is necessary to ensure that the left and right cameras are symmetrical with respect to the central axis of the vehicle, and the distances between the two fisheye cameras and the front of the vehicle and the baseline are consistent. Based on the requirements of viewing angle acquisition, the four-way fisheye camera may be 50 cm above the ground, and especially the left and right cameras need to be kept at the same height. Figure 2 As shown, the angle between the sight direction of the four-way fisheye camera device and the vertical ground direction is approximately between 40-45 degrees, and does not exceed 45 degrees at most, ensuring that the car body can be seen in the image captured by the four-way fisheye camera device.
[0055] The target detection device can detect targets based on images captured by multiple cameras, including parking spaces, people, obstacles, lane lines, etc., to understand the surrounding environment of the car. For example, target detection is performed using technologies such as neural networks or computer vision.
[0056] It should be noted here that the system to which the target detection method of the present application is applicable is not limited to including four-way fisheye camera devices, and can include any number of camera devices of any type, subject to the technical solution that can implement the present application, and no restriction is imposed on the composition of the system. In the following embodiments of the present application, for the convenience of explanation, only a multi-way fisheye camera device is used as an example for explanation.
[0057] The target detection method provided in this application is described below.
[0058] Figure 3 This is a flow chart of an embodiment of the target detection method of the present application, such as Figure 3 As shown, the method of this embodiment can be executed by the chip in the above ADAS, and the target detection method may include:
[0059] Step 301: When the first target and the second target are the same target, a fusion confidence is obtained according to the first confidence and the second confidence.
[0060] The first confidence level is the credibility of the first target in the first image, the second confidence level is the credibility of the second target in the second image, the first image is an image from a first camera device, and the second image is a surround image obtained based on images from multiple cameras. Optionally, the multiple cameras include the first camera device. In another embodiment, the multiple cameras do not include the first camera device.
[0061] In the present application, the first image is an image captured by the first camera device. Further, the plurality of camera devices may be arranged on the vehicle body, but the specific positions are not limited.
[0062] Specifically, multiple cameras are installed around the vehicle body. The camera coverage areas of two or more adjacent cameras may partially overlap. For example, as described in the above embodiment, in the four-way fisheye camera on the vehicle body, each camera can be considered as a first camera, so the first image can come from one of the cameras or from multiple cameras. If the vehicle-mounted camera is a fisheye camera, the image captured is a fisheye image. Since the field of view of the fisheye camera is large, many image features can be captured, so the three-dimensional fisheye image captured can effectively display the features of spatial objects. However, there is a distortion phenomenon in the fisheye camera. The farther away from the imaging center point, the greater the image distortion, which is reflected in the fisheye image as geometric deformation of the object, such as stretching and bending.
[0063] Therefore, an embodiment of the present application also provides a second image. The second image is a surround image obtained by stitching images taken by multiple cameras, and the multiple cameras include the above-mentioned first camera. According to the corresponding image relationship of the overlapping areas in the images of multiple angles taken by the multiple cameras, the top views taken by the four fisheye cameras are stitched by the image fusion stitching algorithm to obtain a surround image. The image stitching may include three main steps: fisheye image distortion correction, top view conversion and image stitching. Each step can obtain a coordinate transformation relationship between corresponding pixels in an input image and an output image, and then finally obtain a pixel correspondence relationship from a single image to a surround image. It should be noted that the prior art can be referred to for the stitching technology and the generation of the surround image, and the present application does not specifically limit the specific stitching method and the generation of the surround image.
[0064] It should be further explained that, due to the presence of multiple cameras, there may be multiple first images including the above-mentioned first target among the multiple images captured by the multiple cameras. For the convenience of explanation, the present application scheme involves obtaining detection results based on a first image and a second image. However, it is known to those skilled in the art that the first image may be multiple images, that is, at least two first images among the multiple images captured by multiple cameras. Furthermore, the embodiments of the present application may also perform target detection based on multiple first images and second images. In this scenario, the processing of multiple first images refers to the processing method of the first image in the embodiments of the present application. For example, multiple first images correspond to multiple first confidences. The processing of the first confidence involved below can be extended to the processing of at least one first confidence among the multiple first confidences.
[0065] Optionally, the first image is a fisheye image. Furthermore, the first image may be obtained by processing a fisheye image obtained from a camera device. For example, in order to correct the geometric deformation of an object in an image, the internal parameters and distortion coefficients of the fisheye camera device may be obtained by calibrating the fisheye camera device, and then combined with the external parameters relative to the calibration object, the fisheye image may be corrected according to the obtained parameters and the fisheye distortion model to obtain a corrected image, and the corrected image may be used as the first image. For specific processing methods, reference may be made to image processing methods in the prior art. The above description of obtaining the first image is only a possible image processing method, and the embodiments of the present application do not make specific limitations thereto.
[0066] In the present application, the first confidence level is the credibility of the first target detected from the first image, and the second confidence level is the credibility of the second target detected from the second image.
[0067] For example, target detection can be performed using technologies such as neural networks or computer vision. The main processing methods in the process of target detection based on computer vision include:
[0068] Image grayscale processing: Images are usually stored in red green blue (RGB) format. In the RGB model of an image, when the values of the three color channels of red, green and blue are the same, the value is called a grayscale value. Therefore, each pixel in a grayscale image only needs one byte to store the grayscale value (also known as intensity value or brightness value), and the grayscale range is 0-255. Image grayscale processing can be used as a preprocessing step for image processing to prepare for subsequent operations such as image segmentation, image recognition and image analysis.
[0069] Gaussian filtering: Gaussian filtering is a linear smoothing filter that is suitable for eliminating Gaussian noise and is widely used in the noise reduction process of image processing. In layman's terms, Gaussian filtering is the process of weighted averaging the entire image. The value of each pixel is obtained by weighted averaging the value of itself and other pixels in the neighborhood. The specific operation is to use a template (or convolution, mask) to scan each pixel in the image, and replace the value of the central pixel of the template with the weighted average gray value of the pixels in the neighborhood determined by the template.
[0070] Edge detection: An edge is a collection of pixels whose grayscale changes sharply. It is the most basic feature of an image. Edge detection is to extract the features of discontinuous parts of an image and determine the area based on closed edges.
[0071] Hough transform: The basic principle of Hough transform is to use the duality of points and lines to transform the curve given in the original image space into a point in the parameter space through the curve expression. In this way, the detection problem of a given curve in the original image is transformed into the problem of finding the peak in the parameter space. In other words, the detection of overall characteristics is transformed into the detection of local characteristics, such as straight lines, ellipses, circles, arcs, etc.
[0072] Target detection based on computer vision includes first graying the input image to make it grayscale, then reducing the noise of the image through Gaussian filtering, extracting discontinuous features in the image through edge detection models, and finally converting the overall features into local features through Hough transform. This target detection method can obtain detection results similar to those of neural networks, for example, drawing a frame around the detected target to obtain the confidence of the target, etc.
[0073] Figure 4A and 4B A schematic diagram of a fisheye image and a corrected image is shown. For example, a neural network can be used to detect the target of an image, and the detection result can be presented in the form of an image. For example, to detect whether there is a parking space in an image taken by a vehicle-mounted camera device, the output image obtained by the neural network can be framed, highlighted, or the like to circle the area detected as a parking space. For example, Figure 5 FIG. 1 shows a schematic diagram of the detection result of the first image (single fisheye image) detecting a parking space through a neural network. Figure 6 The figure shows a schematic diagram of the detection result of the second image (surround view image) detecting a parking space through a neural network, and the box in the figure is the detected parking space. In addition, target detection can also give a description of the degree of credibility of the detection result, that is, the confidence of the target, and its value range is between 0-1. Exemplarily, by performing target (parking space) detection on the first image through, for example, neural network technology, 2 parking spaces are detected, one of which has a confidence of 0.3 and the other has a confidence of 0.6, indicating that the latter is more likely to be a parking space than the former in real life, and the former may mistakenly detect other objects as parking spaces.
[0074] It should be noted that in addition to the above-mentioned neural network-based target detection method and computer vision-based target detection method, the present application may also use other target detection methods for detecting the first target and the second target, and no specific limitation is made to this.
[0075] In the present application, multiple methods can be used to determine whether the first target and the second target are the same target.
[0076] In one possible implementation, whether the first target and the second target are the same target is determined based on the image overlap between the first target and the second target. When the image overlap between the first target and the second target is greater than a fourth threshold, it is determined that the first target and the second target are the same target. The image overlap between the first target and the second target can be obtained based on the pixel correspondence between the first image and the second image.
[0077] In another possible implementation, the judgment is made based on the pixel distance between the center point of the first target and the center point of the second target. Specifically, the second target is converted to the second image according to the pixel correspondence relationship to obtain its center point, and then the pixel distance between the center point and the center point of the second target is calculated, and the pixel distance is compared with the set pixel distance threshold to obtain a judgment result of whether the first target and the second target are the same target. For example, the resolution of the second image is 600×600, the pixel distance threshold is 30, the center point of the first target is (400,500), and the center point of the second target is (420,520). The pixel distance between the two center points is calculated to be 28, which is less than 30. Therefore, it can be judged that the first target and the second target are the same target.
[0078] It should be noted that the above two methods are exemplary illustrations, and the present application does not specifically limit the method for determining whether the first target and the second target are the same target.
[0079] For example, Figure 7 A schematic diagram of the detection results of parking spaces detected in the first image (single fisheye image) is shown, wherein the dotted box is a parking line mark for marking parking spaces. When the parking line mark in the dotted box is detected, it can be considered that there is a parking space there. The solid box is a bounding box (BB) that is drawn according to the detected parking line mark to indicate the parking space. Figure 8 The parking space detected in the second image is shown. Figure 7 Schematic diagram of the detection result of converting the parking space detected in the first image (single fisheye image) into the second image (surround view image) according to the pixel correspondence, wherein the dotted box represents the BB of the parking space obtained after conversion based on the parking space detected in the first image, and the solid box represents the BB of the parking space detected in the second image. It can be seen from the figure that the solid box is completely contained in the dotted box, indicating that the two BBs correspond to the same parking space.
[0080] This application sets a threshold value (a fourth threshold value, such as 0.7) for the overlap between two BBs. The overlap can be represented by the Intersection Over Union (IOU). Fig. 9 shows the overlapping area of two BBs, Fig.10The joint area of the two BBs is shown, IOU = overlapping area / joint area. When the IOU is greater than the fourth threshold, it means that the two BBs correspond to the same parking space. At this time, the fusion confidence can be obtained according to the first confidence and the second confidence. However, if the IOU is less than or equal to the fourth threshold, it means that the two BBs may not correspond to the same parking space, and there is no need to fuse the confidences of the two BBs. At this time, there is no need to obtain the fusion confidence and the target detection is terminated.
[0081] Step 302: Output the target detection result according to the fusion confidence.
[0082] The present application can obtain the fusion confidence based on the comparison result of the first confidence and the first threshold, and / or the comparison result of the second confidence and the second threshold, wherein the first threshold, the second threshold, the fourth threshold, and the third threshold involved later can be determined based on the target detection accuracy, precision and / or recall rate, etc., can also be learned through training, and can also be a pre-configured or defined value. The present application does not specifically limit the method for obtaining various thresholds.
[0083] The target detection result output according to the fusion confidence level in this application includes at least one of the following situations:
[0084] (1) When the first confidence is greater than the first threshold, and the second confidence is greater than the second threshold, determine the fusion confidence according to the preset weight value; determine that the fusion confidence is greater than the third threshold, and output the positioning information of the second target detected in the second image. In an optional implementation, for a scene where there are multiple first images (that is, the multiple first images all contain the first target), there are multiple first confidences corresponding to the multiple first images, and the above condition can also be replaced by "when at least one of the multiple first confidences is greater than the first threshold or all are greater than the first threshold, and the second confidence is greater than the second threshold". It should be noted here that the multiple first images in the optional implementation involved below have the same interpretation as the multiple first images mentioned here, and both involve multiple first images containing the same first target.
[0085] Optionally, a first weight and a second weight are provided, wherein the first weight corresponds to the first confidence and the second weight corresponds to the second confidence. Further, the first weight and the second weight are preset. Specifically, the first confidence is weighted by the first weight, and the second confidence is weighted by the second weight, and the two weighted sums are used to obtain a fusion confidence.
[0086] For example, according to the results of target detection on the first image and the second image, a first weight can be set corresponding to the first confidence, and a second weight can be set corresponding to the second confidence. The first confidence is weighted by the first weight, and the second confidence is weighted by the second weight. The two weights are summed to obtain the fusion confidence. In this case, the first confidence of the first target detected in the first image and the second confidence of the second target detected in the second image are both higher than their respective thresholds, indicating that the first target detected in the first image and the second target detected in the second image are both likely to be real targets. In order to improve the confidence of the final detection result, the fusion confidence is obtained based on the detection result of the second image combined with the detection result of the first image, which can improve the confidence of the detection result of the second image and ultimately improve the credibility of the target. If the fusion confidence is greater than the third threshold, the second target detected in the second image can be directly used as the final detected target, and the positioning information of the target is output, and the positioning information is used to present the surrounding environment of the vehicle.
[0087] (2) When the first confidence is less than or equal to the first threshold, and the second confidence is greater than the second threshold, the fusion confidence is determined to be the second confidence; when the fusion confidence is determined to be greater than the third threshold, the positioning information of the second target detected in the second image is output. In an optional implementation, for a scene with multiple first images, there are multiple first confidences corresponding to the multiple first images, and the above condition can also be replaced by "when all the multiple first confidences are less than or equal to the first threshold, and the second confidence is greater than the second threshold".
[0088] In this case, the first confidence of the first target detected in the first image is less than or equal to the first threshold, and the second confidence of the second target detected in the second image is greater than the second threshold, indicating that the first target detected in the first image is more likely to be a false detection, while the second target detected in the second image is more likely to be a real target, so the second confidence of the second target is used as the fusion confidence. If the fusion confidence is greater than the third threshold, the second target detected in the second image can be used as the final detected target, and the positioning information of the target is output, and the positioning information is used to present the surrounding environment of the vehicle.
[0089] (3) When the first confidence is greater than the first threshold, and the second confidence is less than or equal to the second threshold, the fusion confidence is determined to be the first confidence; when the fusion confidence is determined to be greater than the third threshold, the positioning information of the first target detected in the first image is output. In an optional implementation, for a scene where there are multiple first images, there are multiple first confidences corresponding to the multiple first images, and the above feature can also be replaced by "when at least one of the multiple first confidences is greater than the first threshold or all are greater than the first threshold, and the second confidence is less than or equal to the second threshold, the fusion confidence is determined to be one of the multiple first confidences. For example, the fusion confidence can be the first confidence with the largest value among the multiple first confidences."
[0090] In this case, the first confidence of the first target detected in the first image is greater than the first threshold, and the second confidence of the second target detected in the second image is less than or equal to the second threshold, indicating that the first target detected in the first image is more likely to be a real target, while the second target detected in the second image is more likely to be a false detection, so the first confidence of the first target is used as the fusion confidence. If the fusion confidence is greater than the third threshold, the first target detected in the first image can be used as the final detected target, and the positioning information of the target is output, and the positioning information is used to present the surrounding environment of the vehicle.
[0091] (4) When the first confidence level is less than or equal to the first threshold, and the second confidence level is less than or equal to the second threshold, it is determined that the fusion confidence level acquisition fails; and a prompt message indicating that the target detection fails is output. Optionally, for a scene with multiple first images, there are multiple first confidence levels corresponding to the multiple first images, and the above condition can also be replaced by "when all of the multiple first confidence levels are less than or equal to the first threshold, and the second confidence level is less than or equal to the second threshold".
[0092] In this case, the first confidence of the first target detected in the first image is less than or equal to the first threshold, and the second confidence of the second target detected in the second image is less than or equal to the second threshold, indicating that the first target detected in the first image and the second target detected in the second image are both likely to be false detections. In this case, it can be determined that the fusion confidence acquisition has failed, and a prompt message of target detection failure is directly output.
[0093] It should be noted that the above four situations of outputting target detection results based on the fusion confidence, the conditions of each situation are an exemplary description. For the special case where the first confidence is equal to the first threshold and the second confidence is equal to the second threshold, the situation to which it should be classified can be based on the specific settings of the first threshold and the second threshold.
[0094] For example, the condition of the above situation (1) may also be that the first confidence is greater than or equal to the first threshold, and the second confidence is greater than or equal to the second threshold, the condition of situation (2) may also be that the first confidence is less than the first threshold, and the second confidence is greater than or equal to the second threshold, the condition of situation (3) may also be that the first confidence is greater than or equal to the first threshold, and the second confidence is less than the second threshold, and the condition of situation (4) may also be that the first confidence is less than the first threshold, and the second confidence is less than the second threshold.
[0095] For another example, the condition of the above situation (1) may also be that the first confidence is greater than the first threshold, and the second confidence is greater than or equal to the second threshold, the condition of situation (2) may also be that the first confidence is less than or equal to the first threshold, and the second confidence is greater than or equal to the second threshold, the condition of situation (3) may also be that the first confidence is greater than the first threshold, and the second confidence is less than the second threshold, and the condition of situation (4) may also be that the first confidence is less than or equal to the first threshold, and the second confidence is less than the second threshold.
[0096] For another example, the condition of the above situation (1) may also be that the first confidence is greater than or equal to the first threshold, and the second confidence is greater than the second threshold, the condition of situation (2) may also be that the first confidence is less than the first threshold, and the second confidence is greater than the second threshold, the condition of situation (3) may also be that the first confidence is greater than or equal to the first threshold, and the second confidence is less than or equal to the second threshold, and the condition of situation (4) may also be that the first confidence is less than the first threshold, and the second confidence is less than or equal to the second threshold.
[0097] The present application ensures the credibility of the detected targets by fusing the confidence of the targets detected from a single image with the confidence of the targets detected from the surround view images, thereby improving the accuracy of target detection.
[0098] Exemplarily, assume that the first confidence is x, the second confidence is y, the first threshold A=0.6, the second threshold B=0.4, the weight of x a=0.2, the weight of y b=0.8, the third threshold is 0.5, and the fourth threshold is 0.7. Fig.11 The process of fusing the confidence of the first image and the second image in the target detection method is shown, including:
[0099] Step 1101: Determine whether the IOU is greater than 0.7.
[0100] IOU is the image overlap between an object detected in the first image and an object detected in the second image, obtained according to the pixel correspondence between the first image and the second image.
[0101] If IOU>0.7, execute step 1102; if IOU≤0.7, execute step 1107 to terminate the confidence fusion process.
[0102] Step 1102: Compare x with A, and compare y with B.
[0103] Step 1103: When x>A and y>B, the fusion confidence z=ax+by.
[0104] For example, x=0.8, y=0.5, z=0.2×0.8+0.8×0.5=0.56. z>0.5, the positioning information of the target detected in the second image is output.
[0105] Step 1104: When x≤A and y>B, z=y.
[0106] For example, x = 0.6, y = 0.5, z = 0.5. Output the positioning information of the target detected in the second image.
[0107] Step 1105: When x>A and y≤B, z=x.
[0108] For example, x = 0.8, y = 0.3, z = 0.8. Output the positioning information of the target detected in the first image.
[0109] Step 1106: When x≤A and y≤B, acquisition of z fails.
[0110] Fig.12 This is a schematic diagram of the structure of an embodiment of the target detection device of the present application, such as Fig.12 As shown, the target detection device of this embodiment can be independently set, integrated in the control device, or implemented by software or a combination of software and hardware. For example, it can be applied to ADAS, specifically implemented as a chip or integrated circuit in the ADAS, or implemented independently of the ADAS; for another example, it can be applied to the control entity of the vehicle, specifically integrated in the control entity or independent of the control entity.
[0111] The device includes: a processing module 1201 and an interface module 1202, wherein the processing module 1201 is configured to obtain a fusion confidence based on a first confidence and a second confidence when a first target and a second target are the same target; wherein the first confidence is the credibility of the first target in the first image, and the second confidence is the credibility of the second target in the second image, the first image is an image from a first camera device, and the second image is a surround view image obtained based on images from multiple cameras, and the multiple cameras include the first camera device; the interface module 1202 is configured to output a target detection result based on the fusion confidence.
[0112] In one possible implementation, the processing module 1201 is configured to determine the fusion confidence according to a preset weight value when the first confidence is greater than a first threshold and the second confidence is greater than a second threshold; and / or, when the first confidence is less than or equal to the first threshold and the second confidence is greater than the second threshold, determine that the fusion confidence is the second confidence.
[0113] In a possible implementation, the processing module 1201 is configured to determine that the fusion confidence is greater than a third threshold; and the interface module 1202 is configured to output positioning information of the second target detected in the second image.
[0114] In a possible implementation, the processing module 1201 is configured to determine that the fusion confidence is the first confidence when the first confidence is greater than the first threshold and the second confidence is less than or equal to the second threshold.
[0115] In a possible implementation, the processing module 1201 is configured to determine that the fusion confidence is greater than a third threshold; and the interface module 1202 is configured to output positioning information of the first target detected in the first image.
[0116] In one possible implementation, the processing module 1201 is configured to determine whether the first target and the second target are the same target based on the image overlap of the first target and the second target; when the image overlap of the first target and the second target is greater than a fourth threshold, it is determined that the first target and the second target are the same target.
[0117] In a possible implementation, the processing module 1201 is configured to acquire the first image from the first camera device; and to stitch images from the multiple camera devices to obtain the second image.
[0118] The device of this embodiment can be used to perform Figure 3 Or the technical solution of the method embodiment shown in 11, its implementation principle and technical effect are similar, and will not be repeated here.
[0119] In a possible implementation, the present application provides a computer-readable storage medium, including a computer program, which, when executed on a computer, enables the computer to perform the above Figure 3-11 The method of any of the illustrated method embodiments.
[0120] In a possible implementation, the present application provides a computer program, which, when executed by a computer, implements the above Figure 3-11 The method of any of the illustrated method embodiments.
[0121] In one possible implementation, the present application provides a device, which can be independently configured, integrated in a control device, or implemented through software or a combination of software and hardware. For example, it can be applied to ADAS, specifically implemented as a chip or integrated circuit in the ADAS, or implemented independently of the ADAS; for another example, it can be applied to a control entity of a vehicle, specifically integrated in the control entity or independent of the control entity. The device includes a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to implement the above-mentioned Figure 3-11 The method of any of the illustrated method embodiments.
[0122] See also Fig.13 , Fig.13 The target detection device 1300 may be applied to ADAS or may be a control entity of a vehicle, and includes a processor 1301 and a transceiver 1302 .
[0123] Optionally, the target detection device 1300 further includes a memory 1303. The processor 1301, the transceiver 1302 and the memory 1303 may communicate with each other through an internal connection path to transmit control signals and / or data signals.
[0124] The memory 1303 is used to store computer programs. The processor 1301 is used to execute the computer programs stored in the memory 1303, so as to realize the functions of the target detection device 1300 in the above device embodiment.
[0125] Specifically, the processor 1301 may be used to execute an apparatus embodiment (eg, Fig.12 ) is used to perform the operations and / or processing performed by the fusion module 1202 and the output module 1203, while the transceiver 1302 is used to perform the operations and / or processing performed by the acquisition module 1201.
[0126] Optionally, the memory 1303 may also be integrated into the processor 1301 , or independent of the processor 1301 .
[0127] Optionally, the target detection device 1300 may further include a power supply 1304 for providing power to various devices or circuits.
[0128] In addition, in order to make the function more complete, the target detection device 1300 may also include one or more of an input unit 1305, a display unit 1306 (which may also be considered as an output unit), an audio circuit 1307, a camera 1308, and a sensor 1309. The audio circuit may also include a speaker 13071, a microphone 13072, etc., which will not be repeated.
[0129] The processor and memory mentioned in the above embodiments may be located on an integrated circuit or chip, and the processor has image processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The processor may be a general-purpose processor, a digital signal processor (digital signal processor, DSP), and the integrated circuit or chip may be an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware coding processor, or executed by a combination of hardware and software modules in the coding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0130] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0131] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0133] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0136] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.
[0137] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A target detection method, characterized in that: include: When the first target and the second target are the same target, a fusion confidence is obtained according to a first confidence and a second confidence; wherein the first confidence is a credibility of the first target in the first image, the second confidence is a credibility of the second target in the second image, the first image is an image from a first camera device, and the second image is a surround image obtained according to images from multiple cameras, and the multiple cameras include the first camera device; Outputting a target detection result according to the fusion confidence; When the first confidence is greater than a first threshold and the second confidence is greater than a second threshold, determining the fusion confidence according to a preset weight value; and / or, when the first confidence is less than or equal to the first threshold and the second confidence is greater than the second threshold, determining the fusion confidence to be the second confidence; Determining that the fusion confidence is greater than a third threshold; outputting positioning information of the second target detected in the second image; and / or, When the first confidence is greater than the first threshold and the second confidence is less than or equal to the second threshold, determining the fusion confidence to be the first confidence; Determining that the fusion confidence is greater than a third threshold; outputting positioning information of the first target detected in the first image; and / or, When the first confidence is less than or equal to the first threshold, and the second confidence is less than or equal to the second threshold, determining that the fusion confidence acquisition fails; Outputs prompt information for target detection failure.
2. The method according to claim 1, characterized in that Also includes: Determining whether the first target and the second target are the same target according to the image overlap degree of the first target and the second target; When the image overlap degree of the first object and the second object is greater than a fourth threshold, it is determined that the first object and the second object are the same object.
3. The method according to claim 1 or 2, characterized in that: Also includes: Acquire the first image from the first camera device; as well as The second image is obtained by stitching images from the multiple camera devices.
4. A target detection device, characterized in that: include: a processing module configured to, when the first target and the second target are the same target, obtain a fusion confidence according to a first confidence and a second confidence; wherein the first confidence is a credibility of the first target in the first image, the second confidence is a credibility of the second target in the second image, the first image is an image from a first camera device, the second image is a surround view image obtained based on images from multiple cameras, and the multiple cameras include the first camera device; An interface module, configured to output a target detection result according to the fusion confidence; The processing module is configured to determine the fusion confidence according to a preset weight value when the first confidence is greater than a first threshold and the second confidence is greater than a second threshold; and / or, when the first confidence is less than or equal to the first threshold and the second confidence is greater than the second threshold, determine that the fusion confidence is the second confidence; The processing module is configured to determine that the fusion confidence is greater than a third threshold; The interface module is configured to output positioning information of the second target detected in the second image; and / or, The processing module is configured to determine that the fusion confidence is the first confidence when the first confidence is greater than the first threshold and the second confidence is less than or equal to the second threshold; The processing module is configured to determine that the fusion confidence is greater than a third threshold; The interface module is configured to output positioning information of the first target detected in the first image; and / or The processing module is configured to determine that the fusion confidence acquisition fails when the first confidence is less than or equal to the first threshold and the second confidence is less than or equal to the second threshold; The interface module is configured to output prompt information of target detection failure.
5. The device according to claim 4, characterized in that The processing module is configured to determine whether the first target and the second target are the same target based on the image overlap of the first target and the second target; when the image overlap of the first target and the second target is greater than a fourth threshold, determine that the first target and the second target are the same target.
6. The device according to claim 4 or 5, characterized in that The processing module is configured to acquire the first image from the first camera device; and to stitch images from the multiple camera devices to obtain the second image.
7. A target detection device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the device implements the method according to any one of claims 1 to 3.
8. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 3.
9. A computer program product, characterized in that When the computer program product is executed by a computer, the method according to any one of claims 1 to 3 is implemented.
