Panoramic object detection method and device, vehicle and storage medium

Abstract

The present application belongs to the technical field of aided driving. Disclosed are a panoramic object detection method and device, a vehicle and a storage medium. The method of the present application comprises: acquiring scene images collected by all cameras on the periphery of a vehicle; on the basis of all the scene images, generating a panoramic scene image of a bird's-eye view; for every two adjacent scene images, determining an identified object in an image overlapping area between the two scene images, and determining ranging information between the identified object and the vehicle, the ranging information being obtained respectively by two current cameras; for every two adjacent scene images, on the basis of respective ranging fusion weights of the two current cameras, fusing the two pieces of ranging information to obtain unique ranging information between the identified object and the vehicle; and, on the basis of the unique ranging information, generating in the panoramic scene image tagging information of the identified object.

Description

Panoramic target detection method, device, vehicle and storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 202410066056.0 filed on January 16, 2024, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of assisted driving technology, and in particular to a panoramic target detection method, device, vehicle, and storage medium. Background Art

[0004] The surround-view imaging system is designed to enhance driving experience, safety and convenience, and plays a particularly important role in low-speed driving scenarios.

[0005] However, in related technologies, there are some blind spots or dead angles in the detection of panoramic imaging systems. For example, in the stitching area of ​​multiple camera perspectives, the panoramic imaging system may have image misalignment defects. This may cause the target object to disappear on the display screen or its position relative to the vehicle body to be difficult to judge, thereby causing the driver to misjudge the existence and relative distance of the target object, which may affect the timely detection and avoidance of obstacles. Summary of the Invention

[0006] The main purpose of this application is to provide a panoramic target detection method, device, vehicle and storage medium, aiming to solve the problem in related technologies that due to processing defects in image stitching areas, panoramic imaging system detection may have blind spots and dead angles, causing the target object to disappear on the display screen or its position relative to the vehicle body to be difficult to judge.

[0007] To achieve the above-mentioned objectives, in a first aspect, the present application provides a panoramic target detection method, comprising: acquiring scene images captured by cameras around a vehicle; generating a panoramic scene image from a bird's-eye view based on all the scene images; determining, for each two adjacent scene images, an identified target in an image overlap area between the two scene images, and determining ranging information between the identified target and the vehicle obtained by two current cameras, respectively; wherein the two current cameras are cameras corresponding to the two adjacent scene images, respectively, and in a preset fusion weight table, the two current cameras respectively have ranging fusion weights; for each two adjacent scene images, based on the respective ranging fusion weights of the two current cameras, the two ranging information are fused to obtain unique ranging information between the identified target and the vehicle; and based on the unique ranging information, marking information of the identified target is generated in the panoramic scene image.

[0008] In one embodiment, a panoramic scene image from a bird's-eye view is generated based on all scene images, including: performing perspective conversion on the scene image to obtain a bird's-eye view image; for each image overlap area between two adjacent bird's-eye view images, determining the pixel fusion weights of any two adjacent bird's-eye view images from a preset fusion weight table; for each two adjacent bird's-eye view images, based on the two pixel fusion weights, fusing the image overlap areas of the two adjacent bird's-eye view images to obtain fused adjacent bird's-eye view images; and generating a panoramic scene image based on all fused adjacent bird's-eye view images.

[0009] In one embodiment, for every two adjacent scene images, the identified targets in the image overlap area between the two scene images are determined, including: inputting each scene image into the target detection model to obtain the identified targets; the identified targets have classification confidence and positioning confidence; according to the classification confidence and positioning confidence of the target image area of ​​the identified targets, a comprehensive confidence is obtained; and the identified targets whose comprehensive confidence is greater than the preset confidence are output as the final identified targets.

[0010] In one embodiment, the target detection model outputs multiple overlapping detection frames for the same identified target; the step of obtaining a comprehensive confidence based on the classification confidence and positioning confidence of the target image area of ​​the identified target includes: determining the positioning confidence and classification confidence corresponding to each detection frame of the identified target; screening out all detection frames whose positioning confidence of the detection frames is less than a preset positioning confidence threshold to obtain a preliminary screening detection frame set; for each detection frame in the preliminary screening detection frame set, determining the classification result with the largest classification confidence as the target category of the identified target corresponding to the detection frame; merging the detection frames in the preliminary screening detection frame set based on the target category to obtain a merged detection frame set; using a weighted non-maximum algorithm to perform weighted fusion on all the detection frames in the merged detection frame set to obtain a unique detection frame corresponding to the identified target; obtaining the comprehensive confidence based on the classification confidence and positioning confidence of the unique detection frame; the comprehensive confidence = classification confidence × positioning confidence.

[0011] In one embodiment, the identified target whose comprehensive confidence is greater than the preset confidence is output as the final identified target, including: outputting the identified target whose comprehensive confidence is greater than the preset confidence as the identified target; updating the current target detection set based on the identified target; and outputting all identified targets in the current target detection set.

[0012] In one embodiment, for every two adjacent scene images, two ranging information are fused based on the respective ranging fusion weights of the two current cameras to obtain unique ranging information between the identified target and the vehicle, including: if the identified target appears in the overlapping area of ​​the images, the image features corresponding to the identified target are identified in the two scene images respectively; based on the pixel position information of the image features in the binocular imaging system, the binocular ranging information of the identified target is determined; the binocular imaging system is composed of two cameras corresponding to the two scene images; based on the two ranging fusion weights and the binocular fusion weight of the binocular ranging information, the binocular ranging information is fused with the two ranging information to obtain unique ranging information of the identified target.

[0013] In one embodiment, for every two adjacent scene images, the two ranging information are fused based on the ranging fusion weights of the two current cameras to obtain unique ranging information between the identified target and the vehicle, including: obtaining the perception information of the identified target collected by the perception sensor; determining the ranging fusion weight corresponding to the perception sensor from a preset fusion weight table; and fusing the ranging information between the sensors based on the ranging fusion weight to obtain unique ranging information of the identified target.

[0014] In one embodiment, for every two adjacent scene images, after fusing the two ranging information based on the respective ranging fusion weights of the two current cameras to obtain unique ranging information between the identified target and the vehicle, the method further includes: determining the alarm level of the identified target based on the unique ranging information of each identified target; determining the alarm target corresponding to the highest alarm level from all identified targets; determining the target camera to which the alarm target belongs; and displaying the scene image of the target camera.

[0015] In a second aspect, to achieve the above-mentioned purpose, the present application further provides a panoramic target detection device, which includes: an acquisition module, which acquires scene images captured by each camera around the vehicle; a panoramic image generation module, which generates a panoramic scene image based on all scene images; a ranging fusion weight calculation module, which determines, for each two adjacent scene images, an identified target in the image overlap area between the two scene images, and determines the ranging information between the identified target and the vehicle obtained by the two current cameras respectively; wherein the two current cameras are cameras corresponding to the two adjacent scene images, and in a preset fusion weight table, the two current cameras respectively have ranging fusion weights; the fusion module, which, for each two adjacent scene images, fuses the two ranging information based on the respective ranging fusion weights of the two current cameras to obtain unique ranging information between the identified target and the vehicle; and the identification module, which generates marking information of the identified target in the panoramic scene image based on the unique ranging information.

[0016] On the third aspect, in order to achieve the above-mentioned purpose, the present application continues to provide a vehicle, including: a processor, a memory, and a panoramic target detection program stored in the memory, and the panoramic target detection program implements the steps of the above-mentioned panoramic target detection method when run by the processor.

[0017] Fourthly, in order to achieve the above-mentioned purpose, the present application continues to provide a computer-readable storage medium, on which a panoramic target detection program is stored. When the panoramic target detection program is executed by a processor, the above-mentioned panoramic target detection method is implemented.

[0018] The present application performs correlation processing on scene images obtained by a vehicle-body camera to generate a 360° bird's-eye panoramic image around the vehicle, and for an identified target in the image overlap area of ​​two adjacent scene images, determines the ranging information between the identified target and the vehicle obtained by the two cameras respectively. Subsequently, the two ranging information are fused according to the ranging fusion weights preset in a fusion weight table to obtain more accurate unique ranging information between the target and the vehicle. Subsequently, based on the unique ranging information, the position information of the target relative to the vehicle body can be calculated. Based on the position information, corresponding marking information of the identified target can be generated in the generated panoramic scene image.

[0019] This embodiment enables the acquisition of 360° bird's-eye view information around the vehicle, and accurately measures information about objects around the vehicle using range fusion weights, enabling the annotation of identified objects within the panoramic scene image. This annotation of identified objects within the panoramic scene image avoids the problem in traditional panoramic image systems where, due to image misalignment during image stitching, objects disappear from the display or their relative position relative to the vehicle becomes difficult to determine. The marking information within the panoramic scene image allows the driver to accurately understand the presence of obstacles around the vehicle, enabling timely avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 is a schematic structural diagram of the vehicle of the present application;

[0021] FIG2 is a flow chart of a first embodiment of a panoramic target detection method of the present application;

[0022] FIG3 is a schematic diagram of a detailed flow chart of step S200 of the first embodiment of the panoramic target detection method of the present application;

[0023] FIG4 is a flow chart of a second embodiment of a panoramic target detection method of the present application;

[0024] FIG5 is a flow chart of step S330 of the second embodiment of the panoramic target detection method of the present application;

[0025] FIG6 is a flowchart of a third embodiment of a panoramic target detection method of the present application;

[0026] FIG7 is a schematic diagram of the binocular imaging system of the present application;

[0027] FIG8 is a flowchart of a fourth embodiment of a panoramic target detection method of the present application;

[0028] FIG9 is a flowchart of a fifth embodiment of a panoramic target detection method of the present application;

[0029] FIG10 is a schematic diagram of the first structure of an example of this application;

[0030] FIG11 is a schematic diagram of a second structure of an example of the present application;

[0031] FIG12 is a schematic diagram of the third structure of the example of this application.

[0032] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0033] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0034] The surround-view imaging system is designed to enhance driving experience, safety and convenience, and plays a particularly important role in low-speed driving scenarios.

[0035] However, in related technologies, the detection of panoramic imaging systems may have some blind spots or dead angles. For example, in the stitching area of ​​multiple camera perspectives, the panoramic imaging system may have image misalignment defects. This may cause the target object to disappear on the display screen or its position relative to the vehicle body to be difficult to judge, thereby causing the driver to misjudge the existence and relative distance of the target object, which may affect the timely detection and avoidance of obstacles.

[0036] The following describes the panoramic target detection method, device, vehicle, and storage medium used in the implementation of the present invention.

[0037] Refer to Figure 1, which is a structural diagram of a vehicle in the hardware operating environment involved in an embodiment of the present application.

[0038] As shown in Figure 1, the vehicle may include: a processor 1001, such as a CPU, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a voice pickup module, such as a microphone array, etc. The user interface 1003 may also be a display screen (Display), an input unit such as a keyboard (Keyboard), etc. The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0039] The vehicle may further include a network interface 1004, which may include a standard wired interface or a wireless interface (such as a WI-FI interface). In one embodiment, the vehicle may further include RF (Radio Frequency) circuits, sensors, audio circuits, WIFI modules, and the like.

[0040] Those skilled in the art will appreciate that the vehicle structure shown in FIG1 does not limit the vehicle and may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components.

[0041] Based on the above vehicle hardware structure but not limited to the above hardware structure, the present application provides a first embodiment of a panoramic target detection method. Referring to FIG2 , FIG2 shows a flow chart of the first embodiment of the panoramic target detection method of the present application.

[0042] It should be noted that although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in an order different from that shown or described here.

[0043] In this embodiment, the panoramic target detection method includes:

[0044] Step S100: Acquire scene images captured by cameras around the vehicle.

[0045] Step S200: Generate a panoramic scene image from a bird's-eye view based on all scene images.

[0046] In this embodiment, cameras are arranged around the vehicle, and each camera can collect scene information within the camera's field of view to generate a scene image.

[0047] Compared to other cameras, fisheye cameras can obtain scene information with a wider viewing angle. In this embodiment, taking fisheye cameras as an example, four fisheye cameras are arranged in the front, back, left, and right directions of the vehicle to ensure that scene images of 360° scene information around the vehicle can be obtained.

[0048] The viewing angle of the scene image captured by the fisheye camera is usually a horizontal viewing angle, while the bird's-eye view image can provide a more three-dimensional scene observation angle compared to the horizontal viewing angle, helping the driver and the vehicle to more comprehensively understand the information around the vehicle. Therefore, in this embodiment, the captured scene image is subjected to corresponding perspective transformation processing and image stitching operations to generate a panoramic scene image from a bird's-eye view that contains 360° scene information around the vehicle.

[0049] As shown in FIG3 , as a specific implementation, step S200 includes:

[0050] Step S210 , performing perspective conversion on the scene image to obtain a bird's-eye view image.

[0051] In this embodiment, the scene image from the fisheye camera is dedistorted, and based on the dedistorted horizontal perspective scene image, the horizontal perspective scene image is transformed into a bird's-eye view image by using the homography matrix for converting between the horizontal perspective and the bird's-eye view.

[0052] In a specific example, a horizontal scene image with a calibration cloth can be first obtained through a fisheye camera installed on a vehicle, wherein the calibration cloths need to be placed at the four corners of the vehicle, and adjacent calibration cloths need to be kept on the same horizontal line. After obtaining the horizontal scene image, a dedistortion operation is performed on the horizontal scene image, and feature points of the calibration cloth are selected on the dedistorted horizontal image, such as the corner points of the calibration cloth and other feature-clear points. At the same time, the feature points of the calibration cloth in the undistorted scene image under the bird's-eye view corresponding to the horizontal view are found. According to the spatial relationship between the corresponding feature points in the dedistorted horizontal view image and the undistorted bird's-eye view image, the homography matrix for the conversion between the horizontal view image and the bird's-eye view is calculated. Each fisheye camera on the vehicle corresponds to a homography matrix, and the calculated homography matrix is ​​saved for use in the bird's-eye view conversion of the scene picture taken by the subsequent fisheye camera. The specific application method is shown in Formula 1:

[0053] in, Represents the homogeneous coordinates of the corresponding points in the horizontal perspective image taken by a fisheye camera on the vehicle, and H represents the homography matrix corresponding to the camera. Represents the homogeneous coordinates of the corresponding points in the bird's-eye view image obtained after the horizontal view image is transformed into a bird's-eye view image.

[0054] In the example, the horizontal perspective image can be dedistorted by pixel fitting. The known horizontal perspective image is a distorted fisheye image. The process of dedistorting the horizontal perspective image is the process of finding the mapping relationship between the horizontal perspective image and each point in the dedistorted horizontal perspective image. The fisheye camera on the vehicle can be used to shoot calibration objects with known three-dimensional coordinates at various angles, and their corresponding two-dimensional coordinates in the image can be recorded. The fisheye camera is calibrated using the known three-dimensional coordinates and the corresponding two-dimensional coordinates in the captured image. During the camera calibration process, the internal parameters of the camera (such as focal length, principal point position) are estimated, and the distortion fitting coefficient is also estimated. Since the radial distortion of the camera is usually nonlinear, an optimization algorithm (such as the least squares method) is used during the camera calibration process to perform nonlinear optimization on the distortion fitting coefficient to find the optimal distortion fitting coefficient and record it. In practical applications, the specific dedistortion method is shown in Formula 2: x′=x(1+k1r 2 +k2r 4 +k3r 6 +k4r 8 ); y′=y(1+k1r 2 +k2r 4 +k3r 6 +k4r 8 );

[0055] In Formula 2, (x', y') represents the normalized coordinates of the corresponding point in the horizontal view image before dedistortion, (x, y) represents the normalized image coordinates of the corresponding point in the undistorted horizontal view image, k1, k2, k3, and k4 represent the distortion fitting coefficients, and r represents the distance from the corresponding point in the undistorted horizontal view image to the image center.

[0056] By transforming the perspective of scene images captured by the fisheye camera, stereoscopic scene observation is achieved from a horizontal perspective to a bird's-eye view, providing more comprehensive surrounding information for the driver and vehicle. Through dedistortion processing and bird's-eye view conversion, the distorted scene image around the vehicle is successfully converted to a distortion-free bird's-eye view, improving the image's geometric accuracy and information restoration.

[0057] Step S220 : For each image overlap region between two adjacent bird's-eye view images, the pixel fusion weights of any two adjacent bird's-eye view images are determined from a preset fusion weight table.

[0058] In this embodiment, the overlap region between the bird's-eye view images is the overlapped portion of the bird's-eye view images corresponding to the viewing angles of two adjacent fisheye cameras on the vehicle. The pixel fusion weight represents the weight of each pixel within the overlap region when the bird's-eye view images are stitched together.

[0059] After performing a top-down transformation on the horizontal perspective images captured by two adjacent fisheye cameras to obtain corresponding bird's-eye view images, the overlapping portions of the bird's-eye view images are identified, and the weight percentages of the corresponding pixels in the overlapping portions are determined from a fusion weight table. The fusion weight table records the fusion weight percentages of each pixel in the overlapping region of the bird's-eye view images corresponding to each fisheye camera on the vehicle. In this embodiment, the sum of the pixel fusion weights corresponding to the pixels in the overlapping region of two adjacent bird's-eye view images is 1. The weight percentage of each bird's-eye view image is determined based on the fusion effect. In one specific example, for the overlapping region of two adjacent bird's-eye view images, the pixel fusion weights of the corresponding pixels in one bird's-eye view image are incremented from 0 to 1, while the fusion weights of the corresponding pixels in the other bird's-eye view image are decremented from 1 to 0. The fusion weight with the best statistical fusion effect is recorded in the fusion weight table for subsequent image fusion processing.

[0060] By constructing and recording a pixel fusion weight table, we achieve optimized fusion of the bird's-eye view images captured by adjacent fisheye cameras in the overlapping area. By gradually adjusting the pixel fusion weights within the overlapping area of ​​adjacent bird's-eye view images, we determine the optimal fusion effect, ensuring a natural and seamless transition within the overlapping area.

[0061] Step S230 : For every two adjacent bird's-eye view images, based on two pixel fusion weights, performing fusion processing on the image overlap areas of the two adjacent bird's-eye view images to obtain fused adjacent bird's-eye view images.

[0062] In this embodiment, for every two adjacent bird's-eye view images, the pixel fusion weights corresponding to the pixels in the two images are used to perform fusion processing on the overlapping areas between the images.

[0063] According to the pixel fusion weights in the predetermined fusion weight table, weighted fusion is performed on the pixels in the overlapping area of ​​adjacent bird's-eye view images.

[0064] Step S240 : generating a panoramic scene image from a bird's-eye view based on all the fused adjacent bird's-eye view images.

[0065] In this embodiment, an overall image formed by combining all the fused adjacent bird's-eye view images is a panoramic scene image.

[0066] The resulting panoramic scene image reflects the overall situation of the vehicle's surroundings, providing a more comprehensive information basis for subsequent object detection. It also provides drivers with more detailed and accurate environmental awareness, ensuring safe driving in complex road conditions.

[0067] Step S300 : for every two adjacent scene images, determining the recognized target in the image overlap area between the two scene images, and determining the distance measurement information between the recognized target and the vehicle obtained by the two current cameras respectively.

[0068] The two current cameras are cameras corresponding to two adjacent scene images respectively, and in the preset fusion weight table, the two current cameras respectively have ranging fusion weights.

[0069] Step S400 : For every two adjacent scene images, based on the respective ranging fusion weights of the two current cameras, the two ranging information are fused to obtain unique ranging information between the identified target and the vehicle.

[0070] In this embodiment, a recognized target is an object identified in a scene image, such as a person or dog in the scene image that may affect the vehicle's travel. Scene images captured by two cameras at the same time may contain the same recognized target in the overlapping area of ​​the images.

[0071] The distance information represents the measured distance between the identified object and the vehicle in world coordinates. Due to camera errors, the distance information calculated for the identified object may be different depending on the scene image captured by different cameras.

[0072] The distance fusion weight represents the weighted contribution of the distance information between the identified target and the vehicle in the scene image captured by each camera. This embodiment performs distance detection on each identified target in the scene image, determines the corresponding distance information, and then weights this distance information to obtain more accurate, unique distance information.

[0073] By calculating the ratio of world coordinates to pixel coordinates, we can calculate the distance between the identified target and the camera in world coordinates by calculating the pixel distance of the identified target in the scene image. For details, see Formula 3:

[0074] In formula 3, l world is the world coordinate of the identified target, l pixel is the pixel coordinate of the identified target in the scene image, Δdistance pixel is the pixel distance of the identified target in the scene image, Δdistance world The distance information of the identified target in world coordinates.

[0075] After calculating the distance information between the identified target and the vehicle obtained by the two current cameras respectively, the two distance information are weightedly fused according to the distance fusion weights of the two cameras.

[0076] In this embodiment, the ranging fusion weight for each camera is determined by the ranging error of the cameras in the image fusion area. Due to algorithm errors, camera performance, and other factors, the calculated ranging information between the identified target and the camera may differ from the actual distance. To do this, the distance information between two adjacent cameras and the same target at different locations is calculated separately. This distance information is then compared with the actual distance to determine the error distribution for each of the two adjacent cameras. Based on this error distribution, the ranging fusion weight for each of the two adjacent cameras is determined and recorded in a weight fusion table, facilitating the determination of unique ranging information during subsequent actual target recognition.

[0077] The distance fusion weights described above account for the effects of camera image distortion. Within the camera's detection area, the same target can be located at different locations in the image, and distance errors can be affected by image distortion. Image edges are typically more severely distorted, and accordingly, target ranging errors can be increased. By comparing the target's true world coordinate position with the coordinate position obtained through image ranging, an error distribution relationship between the pixel region and the target's true ranging information can be established. For the same target within the stitching area, different cameras may detect different target ranging information. This error distribution, combined with the target's motion characteristics within the time series, can be used to determine whether it is a unique target. Based on this, corresponding fusion coefficients can be assigned according to the aforementioned fusion weights, ultimately yielding accurate target labeling information.

[0078] In a specific example, the method for determining the ranging fusion weight is shown in Formula 4:

[0079] Among them, w i is the ranging fusion weight of the i-th camera, δ i It represents the variance of the ranging errors between the i-th camera and the targets at different positions in the image fusion area. It can be understood that the variance of the ranging errors can represent the error distribution of the camera.

[0080] This embodiment uses weighted fusion to combine the ranging information from each camera in the overlapping area of ​​view between two adjacent cameras, based on the distance fusion weights of the distance information between each camera and the identified target. This method can provide more accurate and unique distance information between the vehicle and the target, helping to reduce uncertainty caused by single-camera ranging errors and improving overall ranging accuracy.

[0081] Step S500 : generating marking information of the identified target in the panoramic scene image based on the unique ranging information.

[0082] After determining the unique ranging information between the identified target and the vehicle, the position coordinate information of the identified target relative to the vehicle can be determined based on the unique ranging information. For example, in the world coordinate system (centered on the vehicle), based on the unique ranging information of the target object and the angle between the identified target and the vehicle, the position coordinates of the identified target in the world coordinate system can be determined. Based on the conversion relationship between the world coordinate system and the image coordinate system of the panoramic scene image, the position information corresponding to the identified target in the panoramic scene image is determined, and marking information of the identified target is generated at the corresponding position. The marking information allows the vehicle driver to intuitively see the relative position and relative distance between the identified target and the vehicle.

[0083] This application achieves the acquisition of 360° bird's-eye view scene information around the vehicle, and accurately measures the information of target objects around the vehicle through ranging fusion weights, thereby achieving the annotation of identified targets in panoramic scene images. By annotating identified targets in panoramic scene images, it can avoid the problem of traditional panoramic image systems where the target disappears from the display screen or its position relative to the vehicle body is difficult to determine due to image misalignment during image stitching. Vehicle drivers can accurately understand the situation of obstacles around the vehicle based on the marking information in the panoramic scene image, thereby achieving timely avoidance.

[0084] 4 , which shows a flow chart of a second embodiment of a panoramic target detection method of the present application, in this embodiment, step S300 includes:

[0085] In step S310 , each scene image is input into a target detection model to obtain an identified target; the identified target has a classification confidence and a location confidence.

[0086] Step S320 , obtaining a comprehensive confidence level based on the classification confidence level and the positioning confidence level of the target image area of ​​the identified target.

[0087] Step S330: Output the identified target whose comprehensive confidence is greater than the preset confidence as the final identified target.

[0088] In this embodiment, each camera around the vehicle corresponds to a target detection model, and the target detection model can be used to detect identified targets in the scene image captured by the corresponding vehicle camera. Identified targets are obstacles that need to be avoided during driving, such as people, other vehicles, dogs, etc. In this embodiment, the classification confidence represents the confidence of the target detection model in the category to which the identified target belongs. Specifically, for each target identified by the target detection model, the classification confidence measures the probability or confidence level that the target belongs to a specific category. For example, if the model detects that a target object is a vehicle, the classification confidence indicates how likely it is that the area is indeed a vehicle.

[0089] The localization confidence represents the confidence of the object detection model in the accuracy of the location of the identified objects in the image. Specifically, for each object detected by the object detection model, the localization confidence measures how confident the model is in the accuracy of the object's location in the image.

[0090] The comprehensive confidence score represents a comprehensive measure of the object detection model's confidence in both the classification and location of an identified object. The comprehensive confidence score is calculated by multiplying the classification confidence score and the location confidence score for the target image region. The comprehensive confidence score more comprehensively reflects the model's overall confidence in each identified object. A higher comprehensive confidence score indicates greater confidence in the model's classification and location information. Identified objects with a comprehensive confidence score greater than a preset confidence score are considered final output.

[0091] In a specific example, data of the target objects to be identified in actual application scenarios are collected offline, including images under different angles, different lighting, different environments, different backgrounds, etc. The images are annotated and input into machine learning models, neural network models, and other models that can perform image recognition and parsing tasks for training. The prediction information output by model reasoning is compared with the manually annotated verification data set to confirm the convergence status of the model training, and finally the required target detection model is obtained. The target detection model will output the positioning confidence, classification confidence, 2D detection box, and key points of the identified target.

[0092] The object detection model may output multiple overlapping detection boxes for the same identified object. To avoid double counting, these detection boxes need to be filtered. In this embodiment, the Weighted NMS (weighted non-maximum suppression) algorithm is used to perform weighted fusion on each detection box, and the detection box with the highest confidence is selected as the final detection result.

[0093] First, the positioning confidence and classification confidence of each detection frame of the identified target are obtained. The detection frames with a positioning confidence lower than a specific positioning confidence threshold are preliminarily filtered out through the positioning confidence. Then, the classification result with the largest classification confidence is selected as the category of each detection frame after the preliminary screening, and these detection frames are classified and merged.

[0094] The detection frame with the highest confidence in the fixed position of the classified and merged detection frames is then used as the reference frame. The intersection-over-union (IoU) ratio of the other detection frames with this detection frame is calculated. Based on the IoU ratio, a weight factor is determined for each detection frame. The larger the IoU ratio, the higher the weight factor of the detection frame. Based on the weight factor, the detection frames after the classification and merging are weighted and fused together to obtain a unique detection frame to be output for the same identified target. This method can detect and mark all identified targets in a scene image, and each different identified target corresponds to a unique detection frame to be output.

[0095] Each identified target is further screened, and different confidence thresholds are set according to different target types. The positioning confidence and classification confidence of the detection box to be output in each identified target are multiplied to obtain a comprehensive confidence. The comprehensive confidence is used as the standard for determining whether the target is a real target, and the identified targets with a comprehensive confidence higher than the preset confidence are output.

[0096] This embodiment is a process of identifying targets in scene images captured by various vehicle cameras. By introducing comprehensive confidence as a criterion for determining whether the identified targets are correct, it provides a more comprehensive and reliable target perception capability, thereby improving the accuracy and robustness of target recognition.

[0097] In order to verify the detection accuracy and recall rate of the target detection in this example, public datasets and custom scene datasets were used for test comparison, and various performance indicators were analyzed and compared. A dataset of 10,137 pictures in complex scenes was used as the test set, with a total of 39,418 labels. The original target detection model was first used to test the dataset. After statistics, the data in Table 1 was obtained as a reference for the performance indicators of the original algorithm model.

[0098] Next, the target detection model of this example was used to test the same batch of data sets. After statistics, Table 2 was obtained as a reference for the performance indicators of the self-developed algorithm.

[0099] Table 3 is obtained as a reference by comparing the performance indicators of the original algorithm model and the algorithm model provided in this example. It can be seen from Table 3 that compared with the original algorithm model, the detection accuracy and recall rate of the target detection model in this example are improved to varying degrees, which further demonstrates that this embodiment provides a more comprehensive and reliable target perception capability and improves the accuracy of target object recognition.

[0100] As a specific implementation, referring to FIG. 5 , in this embodiment, step S330 includes:

[0101] Step S331: The identified target whose comprehensive confidence is greater than the preset confidence is taken as the final identified target.

[0102] Step S332: Update the current target detection set based on the final identified target.

[0103] Step S333: output all identified targets in the current target detection set.

[0104] In this embodiment, the target detection set is a set of all recognized targets in the previous frame of scene image collected.

[0105] This embodiment is a process for dynamic identification of targets in the scene around the vehicle. After identifying the target in the scene image of the current frame, the identified target with a comprehensive confidence greater than the preset confidence is used as the final identified target. The final identified target identified in the current frame is searched and matched with the identified target in the previous frame in the target detection set, and the target detection set is updated. Then, the targets in the updated target detection set are synchronized with the corresponding annotations in the panoramic scene image.

[0106] In one specific example, the current frame of scene images captured by each vehicle camera is fed into the target detection model to predict each identified target. If the current frame is the first frame, the data related to the identified target is directly saved to the target detection set. The target detection set mainly includes the horizontal and vertical coordinates of the center point of the lower edge of the detection frame of the identified target, as well as the width and length of the detection frame. If the current frame is not the first frame, the identified targets in the current frame are sequentially matched with the identified targets in the previous frame in the target detection set.

[0107] If the identified target in the current frame successfully matches a target identified in the previous frame in the target detection set, the identified target in the current frame is considered to be the same target as the identified target in the previous frame. The Kalman filter algorithm is used to update the state of the identified target in the previous frame. The relevant data of the identified target in the previous frame is weightedly fused with the relevant data of the identified target in the current frame to output more stable and smooth updated data.

[0108] If the recognized target in the current frame cannot find a matching recognized target in the previous frame in the target detection set, the recognized target in the current frame is considered to be a new target and is saved in the target detection set.

[0109] If there is an identified target in the target detection set and no matching identified target in the current frame appears in three consecutive frames, it is considered that the identified target in the target detection set has gone out of the vehicle detection range and is removed from the tracking algorithm list.

[0110] By identifying targets with a comprehensive confidence level above a preset threshold as final recognized targets, the system can promptly and accurately identify targets in the current frame's scene image. Subsequently, through steps such as search and match Kalman filtering and fusion processing, the system dynamically updates the target detection set. This effectively reduces data redundancy and enables smooth tracking of target motion. It also automatically handles newly appearing targets and those that move out of view, improving the system's ability to perceive dynamic targets in the scene surrounding the vehicle and providing more accurate and reliable input information for subsequent processing.

[0111] 6 , which shows a flow chart of a third embodiment of a panoramic target detection method of the present application, in this embodiment, step S400 includes:

[0112] Step S410: If the identified target appears in the image overlap area, image features corresponding to the identified target are identified in the two scene images respectively.

[0113] Step S420 : Determine binocular ranging information of the identified target based on pixel position information of the image features in the binocular imaging system.

[0114] Step S430 : Based on the two ranging fusion weights and the binocular fusion weight of the binocular ranging information, the binocular ranging information is fused with the two ranging information to obtain unique ranging information of the identified target.

[0115] The binocular imaging system consists of two cameras corresponding to two scene images. The binocular fusion weight represents the weight of the binocular ranging information when confirming the unique ranging information.

[0116] In this embodiment, binocular ranging information refers to the distance information of the identified target in the image fusion area obtained by the binocular imaging system. The two cameras on a vehicle with overlapping viewing angles can be considered a binocular imaging system. As shown in Figure 7, based on the binocular imaging principle, the binocular ranging information of the identified target relative to the binocular imaging system can be determined. In Figure 7, Ol and Or are the optical centers of the two adjacent cameras, xl and xr are the pixel positions of the ranging reference point P of the identified target on the s1 imaging plane and the s2 imaging plane, and Z is the binocular ranging information between the ranging reference point P and the binocular imaging system.

[0117] Obtain the distance measurement reference point of the identifiable target in the image overlap area of ​​one scene image. Using a feature matching algorithm, search for the distance measurement reference point that matches the identified target in the other scene image. Based on the pixel position information of these two distance measurement reference points in the corresponding images, the binocular distance measurement information of the identified target relative to the binocular imaging system can be calculated. For details, see Formula 5: Z = f*B / D;

[0118] In Formula 5, D is the parallax between the two cameras in the binocular imaging system, which can be determined by the pixel position information of the ranging reference points corresponding to the two cameras. B is the baseline distance between the two cameras. f is the corrected focal length of the two cameras in the binocular imaging system. Z is the binocular ranging information.

[0119] After confirming the binocular ranging information, the binocular ranging information and the ranging information of the two adjacent cameras will be weightedly fused according to the ranging fusion weights corresponding to the two adjacent cameras and the binocular fusion weights corresponding to the binocular imaging system to calculate the unique ranging information.

[0120] The binocular fusion weight can also be found in the fusion weight table. The binocular fusion weight in the fusion weight table can be determined by calculating the error distribution between the binocular ranging information and the actual ranging information between the binocular imaging system and the target at different positions.

[0121] By combining binocular ranging information with the ranging information of two adjacent cameras, the accuracy and robustness of the overall ranging can be further improved.

[0122] 8 , which shows a flowchart of a fourth embodiment of a panoramic target detection method of the present application, in this embodiment, step S400 includes:

[0123] Step S440: Acquire the identified target perception information collected by the perception sensor.

[0124] Step S450: Determine the perception fusion weight corresponding to the perception sensor from a preset fusion weight table.

[0125] Step S460: Based on the two ranging fusion weights and the perception fusion weight, the two ranging information and the identified target perception information are fused to obtain unique ranging information of the identified target.

[0126] In this embodiment, the perception sensor may be an environmental sensing device such as an ultrasonic radar or millimeter-wave radar. The identified target perception information may be the distance between the identified target and the vehicle body, as detected by the perception sensor, within the overlapping visual angle. The perception fusion weight may be the weight assigned to the visual perception information measured by each perception sensor during fusion.

[0127] The perception fusion weight corresponding to the sensor can be obtained from the fusion weight table. According to the perception fusion weight and the ranging fusion weight, the ranging information and the target perception information are weightedly fused to obtain unique ranging information.

[0128] In a specific example, in an object recognition system composed of n perception sensors, sensors s1, s2, ..., sn detect the same target, and the local estimation errors of any two sensors are independent of each other. In the fusion weight table, the perception fusion weights of each sensor are w1, w2, ..., w n (w1+w2+…+w n =1), and the perception fusion weight is determined by the distribution of ranging errors of each sensor in the overlapping area of ​​view. See Formula 6 for details:

[0129] Among them, w i is the perception fusion weight of the i-th perception sensor, δ i represents the variance of the ranging error between the i-th sensor and the target at different positions in the overlapping area of ​​view. In this formula, δ i It also includes the variance of the ranging errors of the targets at different positions in the overlapping area of ​​the camera’s view, that is, w i The ranging fusion weight is also included.

[0130] After confirming the perception fusion weights corresponding to each sensor, the target perception information (ranging information) measured by n perception sensors (including cameras) can be weighted and fused to obtain the final unique ranging information.

[0131] This embodiment combines the ranging information of the target object obtained by the camera and the perception sensor, and uses the ranging fusion weights and perception fusion weights corresponding to the camera and the perception sensor in the fusion weight table to achieve the fusion of multi-source ranging information for the same target, thereby improving the accuracy and reliability of the target ranging information and reducing the error between the target and the actual distance.

[0132] 9 , which shows a flowchart of a fourth embodiment of a panoramic target detection method of the present application, in this embodiment, after step S400, the following steps are further included:

[0133] Step S600: determining the alarm level of each identified target based on the unique ranging information of each identified target.

[0134] Step S700: Determine the alarm target corresponding to the highest alarm level from all identified targets.

[0135] Step S800: Determine the target camera to which the alarm target belongs.

[0136] Step S900: displaying the scene image of the target camera.

[0137] In this embodiment, the alarm level is divided according to the distance between the identified target and the vehicle body. For example, according to the distance between the target object and the vehicle body, the alarm level is divided from low to high into safety, normal alarm, and danger alarm. When the alarm level is in danger alarm, it means that the identified target is too close to the vehicle body. At this time, the driver can be prompted to pay attention by issuing an alarm signal.

[0138] In this embodiment, the unique ranging information of identified targets is used as the criterion for determining the alarm level. The unique ranging information of all identified targets in the panoramic scene image is detected, and the identified target with unique ranging information that meets the highest alarm level is determined as the alarm target. Subsequently, the target camera that detected the alarm target is determined, and the scene image captured by that camera at the current moment is displayed. The distance information of the identified target from the vehicle is annotated at the corresponding position in the image for the driver's reference.

[0139] This embodiment divides the alarm level based on the unique ranging information of the identified target, further determines the alarm target and displays the corresponding scene image, realizes the accurate detection and alarm prompt of panoramic targets, improves the driver's perception and response ability to the surrounding environment, and thus enhances driving safety.

[0140] In order to enable those skilled in the art to better understand this embodiment, the technical solution of this embodiment is explained below through specific implementation examples in specific application scenarios. The following examples are only used to explain this application and are not used to limit the scope of protection of the claims of this application.

[0141] As shown in FIG10 , FIG10 is a structural diagram of a panoramic target detection system in this example. In this example, the system includes a camera, which is installed on the target vehicle according to the method shown in FIG11 , and the camera transmits the AHD signal to the image capture module.

[0142] The image capture module is used to pass the image frames transmitted by the camera to the subsequent surround view module, calibration module and AI module for different algorithm processing.

[0143] The calibration module configures the parameters of all modules within the system and synchronizes these parameters to each module. The calibration module contains the internal parameters of each camera, the distortion fitting coefficients required for image dedistortion, the weight fusion table required for weight fusion, and the homography matrix information required for overhead transformation.

[0144] During parameter configuration, the calibration module selects the relevant parameters of the valid area within the camera's field of view for configuration. For example, the configuration information related to the camera obtaining scene information far away from the vehicle body will not be configured in the calibration module. This allows other modules to only search for relevant information in the valid area when performing table lookup and other related operations, reducing the algorithm's computational time.

[0145] The stitching module can be used to perform perspective conversion and stitching on the same frame of images from various cameras. After processing, a complete 360° panoramic stitching image is obtained and output to the display.

[0146] The AI ​​model is used to parse the designated targets in the images from each camera and select them, while also calculating the distance information of the designated targets.

[0147] The fusion module uses a weighted fusion algorithm to fuse and associate the same target objects in each image frame in the image stitching area, thereby obtaining more accurate target distance information. At the same time, based on the target distance information, the position of the target object relative to the vehicle body is determined, and the corresponding position in the panoramic scene image of the display screen is marked to generate corresponding marking information.

[0148] As shown in Figure 12, this example uses different alarm levels within the panoramic scene image. The alarm level range can be flexibly configured between 0 and 10 meters, divided into normal and dangerous warning zones. Detected targets are displayed on the display screen with visual tracking icons. Based on the target distance information output by the fusion module, different levels of alarm sounds and color changes in the tracking circle indicate when in different dangerous areas, significantly reducing the burden on the driver to process invalid information.

[0149] The error between the target distance information output by the fusion module and the actual distance is smaller, which not only ensures that when the target is at the edge of the alarm area, there will be no missed alarms or false alarms due to precision loss, thereby improving the accuracy of the alarm, but also ensures that when the target is in the overlapping area of ​​the camera's field of view, it will not switch back and forth between adjacent cameras due to data instability, thereby improving efficiency and safety.

[0150] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The above-described program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The above-described storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0151] Based on the same inventive concept, the present application also provides a panoramic target detection device, which includes: an acquisition module for acquiring scene images captured by cameras around a vehicle; a panoramic image generation module for generating a panoramic scene image from a bird's-eye view based on all scene images; a ranging fusion weight calculation module for determining, for each two adjacent scene images, an identified target in the image overlap area between the two scene images, and determining the ranging information between the identified target and the vehicle obtained by the two current cameras respectively; wherein the two current cameras are cameras corresponding to the two adjacent scene images, and in a preset fusion weight table, the two current cameras respectively have ranging fusion weights; a fusion module for fusing the two ranging information for each two adjacent scene images based on the respective ranging fusion weights of the two current cameras to obtain unique ranging information between the identified target and the vehicle; and an identification module for generating marking information of the identified target in the panoramic scene image based on the unique ranging information.

[0152] The various embodiments of the panoramic target detection device in this embodiment and the technical effects achieved therein can refer to the various implementation methods of the panoramic target detection method in the aforementioned embodiments, and will not be repeated here.

[0153] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The above-described program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The above-described storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0154] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

Claims

1. A panoramic target detection method, wherein, The method includes: Obtaining scene images collected by each camera around the vehicle; Generating a panoramic scene image from a bird's-eye view based on all the scene images; For every two adjacent scene images, determining the recognized targets in the image overlapping area between the two scene images, and determining the ranging information between the recognized targets and the vehicle obtained by two current cameras respectively; wherein, the two current cameras are the cameras corresponding to the two adjacent scene images respectively, and in a preset fusion weight table, the two current cameras each have a ranging fusion weight; For every two adjacent scene images, based on the ranging fusion weights of the two current cameras respectively, fusing the two ranging information to obtain the unique ranging information between the recognized target and the vehicle; Based on the unique ranging information, generating marking information of the recognized target in the panoramic scene image.

2. The panoramic target detection method according to claim 1, wherein, The generating a panoramic scene image from a bird's-eye view based on all the scene images includes: Performing a perspective transformation on the scene images to obtain bird's-eye view images; For the image overlapping area between every two adjacent bird's-eye view images, determining the pixel fusion weights of any two adjacent bird's-eye view images respectively from the preset fusion weight table; For every two adjacent bird's-eye view images, based on the two pixel fusion weights, performing a fusion process on the image overlapping area of the two adjacent bird's-eye view images to obtain the fused adjacent bird's-eye view images; Based on all the fused adjacent bird's-eye view images, generating the panoramic scene image from the bird's-eye view.

3. The panoramic target detection method according to claim 1, wherein, The determining the recognized targets in the image overlapping area between the two scene images for every two adjacent scene images includes: Inputting each of the scene images into a target detection model respectively to obtain recognized targets; the recognized targets have a classification confidence level and a localization confidence level; Obtaining a comprehensive confidence level according to the classification confidence level and the localization confidence level of the target image area of the recognized target; Outputting the recognized targets with the comprehensive confidence level greater than a preset confidence level as the final recognized targets.

4. The panoramic target detection method according to claim 3, wherein, The target detection model outputs multiple overlapping detection frames for the same recognized target; The obtaining a comprehensive confidence level according to the classification confidence level and the localization confidence level of the target image area of the recognized target includes: For each detection frame of the recognized target, determining the localization confidence level and the classification confidence level corresponding to the detection frame; Screening out the detection frames with the localization confidence level less than a preset localization confidence level threshold among all the detection frames to obtain a set of preliminarily screened detection frames; For each detection frame in the set of preliminarily screened detection frames, determining the classification result with the maximum classification confidence level as the target category of the recognized target corresponding to the detection frame; Merging the detection frames in the set of preliminarily screened detection frames based on the target category to obtain a set of merged detection frames; Using a weighted non-maximum suppression algorithm to perform weighted fusion on all the detection frames in the set of merged detection frames to obtain the unique detection frame corresponding to the recognized target; Based on the classification confidence and localization confidence of the unique detection box, obtain the comprehensive confidence; the comprehensive confidence = classification confidence × localization confidence.

5. The panoramic target detection method according to claim 3, wherein, The step of outputting the identified target with the comprehensive confidence greater than the preset confidence as the final identified target includes: Taking the identified target with the comprehensive confidence greater than the preset confidence as the identified target; Updating the current target detection set based on the identified target; Outputting all the identified targets in the current target detection set.

6. The panoramic target detection method according to claim 1, wherein, For every two adjacent scene images, based on the ranging fusion weights of the two current cameras respectively, fuse the two ranging information to obtain the unique ranging information between the identified target and the vehicle, including: If the identified target appears in the image overlapping area, respectively identify the image features corresponding to the identified target in the two scene images; Based on the pixel position information of the binocular imaging system according to the image features, determine the binocular ranging information of the identified target; the binocular imaging system is composed of two cameras corresponding to the two scene images; Based on the two ranging fusion weights and the binocular fusion weight of the binocular ranging information, fuse the binocular ranging information with the two ranging information to obtain the unique ranging information of the identified target.

7. The panoramic target detection method according to claim 1, wherein, For every two adjacent scene images, based on the ranging fusion weights of the two current cameras respectively, fuse the two ranging information to obtain the unique ranging information between the identified target and the vehicle, including: Obtain the identified target perception information collected by the perception sensor; Determine the perception fusion weight corresponding to the perception sensor from the preset fusion weight table; Based on the perception fusion weight and the two ranging fusion weights, fuse the two ranging information and the identified target perception information to obtain the unique ranging information of the identified target.

8. The panoramic target detection method according to claim 1, wherein, After the step of, for every two adjacent scene images, based on the ranging fusion weights of the two current cameras respectively, fuse the two ranging information to obtain the unique ranging information between the identified target and the vehicle, the method further includes: According to the unique ranging information of each identified target, determine the decision level of the identified target, where the decision level includes but is not limited to states such as warning, deceleration, braking, etc.; Determine the alarm target corresponding to the highest risk level from all the identified targets; Determine the vehicle orientation to which the alarm target belongs; Display the scene image of the target camera.

9. A panoramic target detection device, wherein, The panoramic target detection device includes: An acquisition module, configured to acquire scene images collected by each camera around the vehicle; A panoramic image generation module, configured to generate a panoramic scene image with a bird's-eye view based on all the scene images; The ranging fusion weight calculation module is used to, for every two adjacent scene images, identify the recognized targets in the image overlapping area between the two scene images, and determine the ranging information of the recognized targets obtained by two current cameras respectively with respect to the vehicle; wherein, the two current cameras are the cameras respectively corresponding to the two adjacent scene images, and in the preset fusion weight table, the two current cameras respectively have ranging fusion weights. The fusion module is used to, for every two adjacent scene images, fuse the two ranging information based on the respective ranging fusion weights of the two current cameras to obtain the unique ranging information of the recognized target with respect to the vehicle. The recognition module is used to generate the marking information of the recognized target in the panoramic scene image based on the unique ranging information.

10. A vehicle, comprising: A processor, a memory, and a panoramic target detection program stored in the memory, the panoramic target detection program, when run by the processor, implements the steps of the panoramic target detection method according to any one of claims 1-8.

11. A computer-readable storage medium, wherein, The panoramic target detection program is stored on the computer-readable storage medium, and when the panoramic target detection program is executed by the processor, it implements the panoramic target detection method according to any one of claims 1 to 8.

Images