Vehicle surround view warning method, device and apparatus

Abstract

The application discloses a vehicle surround-view early warning method, device and equipment, relates to the technical field of target detection, and is applied to a single surround-view early warning field controller and multiple sensors, and comprises the following steps: collecting images around a vehicle through multiple wide-angle cameras; receiving current chassis height information of the vehicle to adaptively switch current calibration parameters, and performing de-distortion and surround-view splicing on the images around the vehicle according to the current calibration parameters to generate a panoramic surround-view image; identifying target objects in the images around the vehicle by using an AI target detection model, and calculating visual target position information of the target objects by using a pixel point fitting and region division algorithm; performing multi-sensor fusion decision on radar target position information and the visual target position information, and outputting fused target object state information; and based on vehicle state data, the panoramic surround-view image and the target object state information, outputting a current triggered alarm signal and a view switching instruction to a vehicle-mounted display device through a blind area unified early warning rule.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a vehicle surround view warning method, device and equipment. Background Technology

[0002] With the gradual implementation of relevant national regulations for large vehicles in categories M2 / N2 / M3 / N3, commercial vehicle and special vehicle manufacturers urgently need a driver assistance system that can simultaneously meet the blind spot warning requirements of multiple relevant national regulations to improve driving safety and successfully pass regulatory certification. Currently, existing blind spot warning solutions on the market typically use multiple independent controllers to adapt to different regulatory requirements and rely on a single vision or radar sensor for target detection. While some solutions attempt to introduce surround view technology, they are mainly designed for conventional vehicle models and do not fully consider the challenges of field of vision obstruction and adaptation caused by the special structures of special vehicles such as cranes, including the front hook, the protruding spare tire / toolbox at the rear, and the variable chassis height.

[0003] Existing technologies often suffer from the following drawbacks in common surround-view warning systems: multi-controller solutions lead to alarm logic conflicts, vehicle integration is difficult and requires regulatory certification, resulting in high time and economic costs; traditional surround-view stitching cannot adapt to changes in crane chassis height, causing calibration parameter failure, image misalignment, and inaccurate ranging; in single-sensor detection mode, the detection rate of vulnerable road users (VRUs), which are of core concern in regulations, is insufficient and the false alarm rate is high, making it difficult to meet the mandatory requirements of regulations for ranging accuracy (e.g., error ≤15cm) and detection rate (≥99.26%); and there is a lack of unified and scalable sensor fusion and failure degradation mechanisms, making it difficult to achieve complete coverage from regulatory warnings to functional safety redundancy on special vehicle models.

[0004] Therefore, how to simultaneously achieve adaptive chassis height surround view stitching, high-precision AI target detection, and multi-sensor fusion decision-making within a single controller, in order to integrate blind spot warning functions that meet the regulations of multiple relevant countries, has become a pressing technical problem to be solved.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a vehicle surround view warning method, device and equipment, which aims to solve the technical problem that the existing single controller solution lacks chassis height adaptation and multi-sensor fusion, and cannot integrate to meet the target blind spot detection accuracy requirements.

[0007] To achieve the above objectives, this application proposes a vehicle surround-view warning method, applied to a single surround-view warning domain controller and multiple sensors, the method comprising: Images of the vehicle's surroundings are captured by multiple wide-angle cameras, including at least one wide-angle camera covering the forward regulatory blind zone defined by the first target regulation, the rear regulatory blind zone defined by the second target regulation, and the lateral regulatory blind zone defined by the third target regulation. The single surround view warning domain controller receives the vehicle's current chassis height information, adaptively switches the current calibration parameters based on the current chassis height information, and performs distortion correction and surround view stitching on the images around the vehicle according to the current calibration parameters to generate a panoramic surround view image. The AI ​​target detection model pre-trained in the single surround view warning domain controller is used to identify targets in the images around the vehicle, and the visual target position information of the targets is calculated by pixel fitting and region segmentation algorithms. The system receives radar target position information detected by radar sensors, performs multi-sensor fusion decision-making by combining the radar target position information with the visual target position information through the single surround-view warning domain controller, and outputs the fused target status information. Based on vehicle status information, the panoramic surround view image, and the target object status information, conflict arbitration is conducted through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller to determine the target regulatory warning task with the highest priority at present, and output the currently triggered alarm signal and view switching command to the vehicle display device.

[0008] In one embodiment, acquiring images of the area surrounding the vehicle using multiple wide-angle cameras includes: The vehicle collects forward and side-front image data using a first wide-angle camera and a second wide-angle camera installed on both sides of the front of the vehicle. Rear and side rear image data are collected by the third and fourth wide-angle cameras installed on both sides of the rear of the vehicle. High-resolution lateral image data is acquired by fisheye and narrow-angle cameras installed on the left and right sides of the vehicle, and lateral ranging data is acquired by millimeter-wave radar.

[0009] In one embodiment, the step of adaptively switching the current calibration parameters based on the current chassis height information, and performing distortion correction and surround view stitching on the images surrounding the vehicle according to the current calibration parameters to generate a panoramic surround view image includes: Based on the current chassis height information, the current homography matrix, current distortion correction coefficients, and current fusion weight table are obtained by matching the pre-stored calibration parameter groups corresponding to different chassis heights. Based on the current distortion correction coefficients, the image around the vehicle is subjected to distortion correction processing, and a distortion-corrected image is output. Based on the current homography matrix, perspective transformation is performed on the distortion-reduced image to generate a bird's-eye view image; Based on the current fusion weight table, pixel-weighted fusion is performed on the overlapping areas in the bird's-eye view images corresponding to adjacent wide-angle cameras to output a panoramic view image.

[0010] In one embodiment, the step of identifying objects in the image around the vehicle using a pre-trained AI object detection model in the single surround-view warning domain controller includes: The image around the vehicle is input into a pre-trained AI object detection model, which outputs an initial set of detection boxes and the location confidence and classification confidence of each initial detection box. The initial detection box with the highest location confidence is selected as the reference box. The intersection-union ratio (IUU) of the remaining initial detection boxes and the reference box is calculated. The weighting factor of the remaining initial detection boxes is determined based on the IUU. The initial detection boxes are then weighted and fused to obtain a unique detection box. The location confidence of the unique detection box is multiplied by the classification confidence to obtain the comprehensive confidence, and the unique detection box with the comprehensive confidence higher than the preset confidence threshold is determined as a valid detection box; The valid detection boxes are matched with the target list of the current tracking algorithm. For the successfully matched valid detection boxes, the state is updated using the Kalman filter algorithm to output smooth detection boxes. The unmatched valid detection boxes are added as new targets to the target list of the tracking algorithm. Targets that have not been matched for a consecutive preset period are removed from the list to output the finally identified target.

[0011] In one embodiment, the step of calculating the visual target location information of the target object through pixel fitting and region segmentation algorithms includes: Based on the pixel position of the target object in the image around the vehicle, the area where the target object is located is divided into regions, and the divided regions are filtered for effective regions. Within the selected effective area, obtain the pre-defined ratio coefficient between world coordinates and pixel coordinates; Calculate the pixel difference between the pixel coordinates of the target object and the reference pixel coordinates based on the scaling factor; The visual target position information of the target object in the vehicle coordinate system is calculated based on the pixel difference and the scaling factor using a pixel fitting algorithm.

[0012] In one embodiment, the step of performing multi-sensor fusion decision-making on the radar target position information and the visual target position information through the single surround-view warning domain controller, and outputting the fused target state information includes: The radar coordinate information collected by the radar sensor is received, and combined with the installation position of the radar sensor, the radar coordinate information is unified into the vehicle coordinate system to obtain the radar target position information. Based on the visual target location information and the radar target location information, the same target objects are associated and matched, and the detection area type of the associated target object is determined. The detection area type includes single sensor detection area and multi-sensor detection overlap area. If the target object is located in the overlapping area of ​​multi-sensor detection, the existence probability of the target object is fused by combining the detection rate, false alarm rate, target confidence and existence period of each sensor in the overlapping area of ​​multi-sensor detection, and the fused existence probability is obtained. The weighting coefficients are determined based on the detection accuracy variance of each sensor within the overlapping area of ​​the multi-sensor detection, and the target position and velocity information provided by each sensor are weighted and averaged based on the weighting coefficients to obtain the fused position and velocity information. By combining the vehicle's speed and wheel angle, the actual speed of the target object is calculated by compensating the fused position and speed information. The sliding window method is then used to determine the target object as being in motion when the actual speed continuously exceeds the motion judgment threshold. The fusion probability, fusion position and velocity information, and motion state are combined to form the fused target state information.

[0013] In one embodiment, after the step of receiving radar target position information detected by radar sensors, performing multi-sensor fusion decision-making by combining the radar target position information and the visual target position information through the single surround-view warning domain controller, and outputting the fused target state information, the method further includes: Real-time monitoring of the health status indicators of each sensor; When any main sensor fails, the type of the failed sensor and the regulatory blind area it covers are determined based on the health status identifier, and the failed sensor information is output. Based on the information of the failed sensors, the weights of the failed sensors are reset to zero, and the weights of the failed sensors are redistributed to the remaining available sensors covering the same regulatory blind zone area according to a preset confidence level allocation rule, and the adjusted fusion weight set is output. Based on the adjusted fusion weight set, the detection data of the remaining available sensors are fused using the existence probability method, and the downgraded fused target state information is output.

[0014] In one embodiment, the step of determining the highest priority target regulatory warning task based on vehicle status information, the panoramic surround view image, and the target object status information, through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller, and outputting the currently triggered alarm signal and view switching command to the vehicle display device includes: Based on the fused position and speed information and motion state in the target object's state information, and combined with the gear, vehicle speed and turn signal status in the vehicle's state information, the warning trigger conditions of the first target regulation, the second target regulation and the third target regulation are evaluated in parallel, and corresponding regulation warning tasks are generated. When the number of regulatory warning tasks triggered at the same time is greater than or equal to two, a cross-regulatory priority arbitration mechanism is activated to arbitrate the conflict of the simultaneously triggered regulatory warning tasks according to a preset regulatory priority sequence. The regulatory priority sequence is set such that the priority of the backward warning task corresponding to the second target regulation is greater than that of the lateral warning task corresponding to the third target regulation, and the priority of the lateral warning task corresponding to the third target regulation is greater than that of the forward warning task corresponding to the first target regulation. Based on the conflict arbitration results, the highest priority target regulatory warning task is determined, and a graded alarm signal is output according to the danger level threshold corresponding to the target regulatory warning task. A perspective switching command is generated that matches the highest priority target regulatory warning task. The perspective switching command has a higher priority than the perspective switching command triggered by the normal gear position and turn signal status, so as to force the vehicle display device to switch to the forward view, rear view, side view or panoramic view image with the target highlighted corresponding to the graded alarm signal.

[0015] Furthermore, to achieve the above objectives, this application also proposes a vehicle surround view warning device, which includes: The image acquisition module is used to acquire images of the vehicle's surroundings through multiple wide-angle cameras, wherein the multiple wide-angle cameras include at least wide-angle cameras covering the forward regulatory blind spot, the rear regulatory blind spot, and the lateral regulatory blind spot; The image calibration module is used to receive the current chassis height information of the vehicle through a single surround view warning domain controller, adaptively switch the current calibration parameters based on the current chassis height information, and perform distortion correction and surround view stitching on the images around the vehicle according to the current calibration parameters to generate a panoramic surround view image. The target detection module is used to identify targets in the images around the vehicle using the pre-trained AI target detection model in the single surround view warning domain controller, and to calculate the visual target position information of the target through pixel fitting and region segmentation algorithms. The fusion decision module is used to receive radar target position information obtained by radar sensors from the detection of the target, and to perform multi-sensor fusion decision-making on the radar target position information and the visual target position information through the single surround-view warning domain controller, and output the fused target status information. The alarm control module is used to determine the highest priority target regulatory warning task based on vehicle status information, the panoramic surround view image and the target object status information, through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller, and output the currently triggered alarm signal and view switching command to the vehicle display device.

[0016] In addition, to achieve the above objectives, this application also proposes a vehicle surround view warning device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the vehicle surround view warning method as described above.

[0017] This application proposes a vehicle surround view warning method, device, and equipment, applied to a single surround view warning domain controller and multiple sensors. The method includes: acquiring images of the vehicle's surroundings using multiple wide-angle cameras, wherein the multiple wide-angle cameras include at least wide-angle cameras covering the forward, rear, and lateral blind spots; receiving the vehicle's current chassis height information through the single surround view warning domain controller, adaptively switching current calibration parameters based on the current chassis height information, and performing distortion correction and surround view stitching on the images of the vehicle's surroundings according to the current calibration parameters to generate a panoramic surround view image; and using a pre-trained AI object detection model in the single surround view warning domain controller to perform object detection on the images of the vehicle's surroundings. The system identifies targets and calculates their visual position information using pixel fitting and region segmentation algorithms. It receives radar target position information detected by radar sensors and uses the single surround-view warning domain controller to perform multi-sensor fusion decision-making between the radar and visual target position information, outputting the fused target status information. Based on vehicle status information, the panoramic surround-view image, and the target status information, it uses the cross-regulatory priority arbitration mechanism in the blind spot unified warning rules integrated by the single surround-view warning domain controller to arbitrate conflicts, determine the highest priority target regulatory warning task, and output the currently triggered alarm signal and view switching command to the in-vehicle display device. This application integrates multiple wide-angle cameras and radar sensors through a single controller (surround view warning domain controller). It adopts blind zone full coverage and image stitching, AI detection and visual ranging, multi-sensor fusion decision-making, and unified warning and view switching by a single controller. It solves the problems of difficult blind zone coverage for special models, performance degradation due to chassis height changes, high false alarm / missed alarm rate and inaccurate ranging, and complex multi-controller integration conflicts and authentication. It achieves full blind zone coverage without dead angles for special models, chassis height adaptive, and single-controller integration, resulting in cost reduction and efficiency improvement. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the vehicle surround view warning method of this application. Figure 2This is a flowchart illustrating Embodiment 2 of the vehicle surround view warning method of this application; Figure 3 This is a flowchart illustrating Embodiment 3 of the vehicle surround view warning method of this application; Figure 4 This is a flowchart illustrating Embodiment 4 of the vehicle surround view warning method of this application; Figure 5 This is a flowchart illustrating Embodiment 5 of the vehicle surround view warning method of this application; Figure 6 This is a schematic diagram of the module structure of the vehicle surround view warning device according to an embodiment of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the vehicle surround view warning method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] Because existing single-controller solutions lack chassis height adaptation and multi-sensor fusion, they cannot meet the target blind zone detection accuracy requirements.

[0025] This application provides a solution that integrates multiple wide-angle cameras and radar sensors through a single controller (surround view warning domain controller). It adopts blind zone full coverage and image stitching, AI detection and visual ranging, multi-sensor fusion decision-making, and unified warning and view switching by a single controller. It solves the problems of difficult blind zone coverage for special models, performance degradation due to chassis height changes, high false alarm / missed alarm rates and inaccurate ranging, and conflicts and complex authentication of multi-controller integration. It achieves full blind zone coverage without dead angles for special models, chassis height adaptive, and single-controller integration, resulting in cost reduction and efficiency improvement.

[0026] Based on this, embodiments of this application provide a vehicle surround view warning method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the vehicle surround view warning method of this application.

[0027] In this embodiment, the vehicle surround view warning method is applied to a single surround view warning domain controller and multiple sensors, and the method includes steps S10~S50: Step S10: Acquire images around the vehicle using multiple wide-angle cameras, wherein the multiple wide-angle cameras include at least one wide-angle camera covering the forward regulatory blind zone defined by the first target regulation, the rear regulatory blind zone defined by the second target regulation, and the lateral regulatory blind zone defined by the third target regulation.

[0028] It should be noted that, in this embodiment, "multi-channel wide-angle camera" refers to a camera device with a wide field of view capable of covering a large area around the vehicle, typically a fisheye camera, used to capture global visual information around the vehicle; "vehicle surrounding image" refers to image data captured by each wide-angle camera, including the vehicle's surrounding environment, pedestrians, vehicles, and other obstacles; the forward regulatory blind spot defined by the first target regulation specifically refers to the visual blind spot area directly in front of the vehicle that must be covered according to relevant national regulations. The rear regulatory blind spot defined by the second target regulation specifically refers to the visual blind spot area directly behind the vehicle that must be covered according to relevant national regulations. The lateral regulatory blind spot defined by the third target regulation specifically refers to the visual blind spot area on the side (especially the right side) of the vehicle that must be covered according to relevant national regulations. The surround view warning domain controller collects images of the vehicle's surroundings through multiple wide-angle cameras. Its purpose is to achieve all-round, blind-spot-free image acquisition of the vehicle's surrounding environment. Combined with surround view stitching and AI detection, it solves the problem that conventional surround view solutions cannot cover blind spots of special models such as the crane hook at the front of special vehicles and the spare tire / toolbox at the rear, ensuring compliance with the mandatory requirements of multiple relevant national regulations for blind spot coverage.

[0029] Additionally, it should be noted that in one possible implementation, the number and placement of the multiple wide-angle cameras can be flexibly configured according to the specific blind spot distribution characteristics of the vehicle model, and are not limited to a fixed number. For example, for special vehicles, two wide-angle cameras can be placed at the front of the crane to avoid obstruction by the hook, and two cameras can be placed at the rear to avoid obstruction by the toolbox.

[0030] In one specific implementation, this application can use eight fisheye surround-view splicing cameras for image acquisition. The front fisheye camera specifically covers the forward regulatory blind spot, the rear fisheye camera covers the rear regulatory blind spot, and the fisheye cameras on the left and right sides cover the lateral regulatory blind spot. Each camera transmits the acquired images around the vehicle to the surround-view warning domain controller via AHD signals, with the transmission delay controlled within 100ms, thereby achieving 360° monitoring of special vehicle crane models without blind spots.

[0031] Step S20: Receive the vehicle's current chassis height information through the single surround view warning domain controller, adaptively switch the current calibration parameters based on the current chassis height information, and perform distortion correction and surround view stitching on the images around the vehicle according to the current calibration parameters to generate a panoramic surround view image.

[0032] It should be noted that, in this embodiment, the single surround-view warning domain controller refers to a unified central computing unit that integrates image processing, AI inference, fusion decision-making, and warning logic; the current chassis height information refers to real-time data reflecting the current ground clearance of the vehicle chassis, which is usually obtained from feedback by the vehicle chassis height sensor or the vehicle controller; adaptive switching refers to the process by which the system automatically adjusts the corresponding parameters according to changes in chassis height; the current calibration parameters refer to calibration data used for image processing, such as camera intrinsic parameters, homography matrix, and fusion weight table, which are matched with the current chassis height; and distortion correction refers to using the calibration parameters to correct and eliminate image edge distortion generated by wide-angle cameras (such as fisheye cameras). The algorithm processing involves: surround view stitching, which refers to projecting images from multiple cameras after perspective transformation onto the same top-view coordinate system and merging overlapping areas of adjacent images to generate a complete image; panoramic surround view image, which refers to a complete 360-degree bird's-eye view image of the vehicle's surroundings after distortion correction and surround view stitching; and surround view warning domain controller, which adaptively switches the current calibration parameters and performs image processing based on the current chassis height information. This aims to solve the problem of surround view stitching misalignment and inaccurate ranging caused by changes in camera installation height due to the variable chassis height of special vehicles such as cranes. It ensures that surround view stitching is not misaligned under different operating conditions, detection and ranging are accurate, and the warning function is not affected.

[0033] Additionally, it should be noted that in one possible implementation, the system can pre-calibrate and store multiple sets of calibration parameters for different chassis heights. When receiving feedback on height changes, it can directly match and call the corresponding parameter set from the database to achieve millisecond-level fast switching.

[0034] In one specific implementation, when the crane adjusts the chassis height, the surround view warning domain controller receives the current chassis height information sent by the vehicle controller, and adaptively switches from the pre-stored database to the camera intrinsic parameters and homography matrix corresponding to that height as the current calibration parameters. Subsequently, the current calibration parameters are used to perform distortion correction and top-down transformation operations on the images around the vehicle collected by the 8 fisheye cameras, and then the overlapping areas are fused according to the saved fusion weight table of adjacent cameras to finally generate a seamless and misaligned panoramic surround view image.

[0035] Step S30: The pre-trained AI target detection model in the single surround view warning domain controller is used to identify targets in the image around the vehicle, and the visual target position information of the targets is calculated by pixel fitting and region segmentation algorithms.

[0036] It should be noted that, in this embodiment, the pre-trained AI target detection model refers to a deep learning model, such as a neural network model, that is trained offline using a large amount of labeled data and capable of feature extraction, classification, and localization of specific targets in an image; target recognition refers to the process of using an AI model to parse and select specified targets (such as pedestrians, two-wheeled vehicles, and other vulnerable road users) from an image; pixel fitting refers to a method of calculating the physical position of a target by establishing a mapping relationship between pixel coordinates and world coordinates and using a fitting algorithm; region segmentation algorithm refers to a strategy of dividing the image into different regions based on differences in ranging accuracy and selecting effective ranging regions to optimize calculations; visual target position information refers to the position and distance information of the target in the vehicle coordinate system calculated by a visual algorithm. This embodiment utilizes AI models and specific algorithms to calculate visual target location information, aiming to overcome the ranging error problem caused by fisheye camera distortion, accurately identify vulnerable road users (VRUs), and improve the ranging accuracy of the fusion area by retaining reliable lateral distance information in the absence of depth information, thus meeting the implicit requirements of regulations for target positioning accuracy. It fully leverages the centralized processing advantages of a single computing platform to accurately extract key targets within regulatory blind spots. Addressing the pain points of large edge distortion and high long-distance ranging errors in wide-angle images, it uses region segmentation to eliminate invalid distortion areas, and then calculates high-precision visual target location information through pixel-level fitting. This achieves a balance between the high recall rate of AI detection and the high accuracy of location calculation, meeting the stringent requirements of regulations for the accuracy of warning locations.

[0037] Additionally, it should be noted that in one possible implementation, in order to reduce the false alarm rate of the AI ​​algorithm, the system sets different comprehensive confidence thresholds for different types of obstacles, and sets the starting period threshold of the tracking algorithm in conjunction with the continuous false detection period of the target, so as to filter out falsely reported targets such as flags and trash cans.

[0038] Step S40: Receive radar target position information detected by radar sensors, and use the single surround-view warning domain controller to perform multi-sensor fusion decision-making on the radar target position data and the visual target position information, and output the fused target status information.

[0039] It should be noted that in this embodiment, radar target position information refers to the distance, speed, and orientation of the target in the vehicle coordinate system detected by the radar sensor; multi-sensor fusion decision refers to the process of associating and complementing the detection information from visual sensors with that from radar and other sensors to comprehensively judge the probability of the target's existence and its motion state; the fused target state information refers to the more accurate and stable comprehensive attribute information such as target distance, speed, type, and motion / stationary state obtained after multi-source data fusion. This embodiment performs multi-sensor fusion decision-making on radar target position information and visual target position information, achieving complementary advantages within a single surround-view warning domain controller. This aims to reduce the false alarm rate and missed alarm rate of a single sensor, improve the accuracy, stability, and ranging precision of target detection, and, in the event of a sensor failure, supplement blind spot detection capabilities through a redundancy mechanism to ensure uninterrupted core regulatory warning functions.

[0040] Additionally, it should be noted that in one possible implementation, the multi-sensor fusion decision-making not only includes existence probability fusion and position information fusion based on variance and error correction weight allocation, but also a sensor failure degradation mechanism. When a failure or data loss of the main sensor is detected, the system dynamically increases the weight coefficient of the auxiliary sensor to upgrade it to the main detection source, and combines the last effective target state information with the vehicle kinematic parameters to perform short-term prediction compensation to eliminate the detection blind time during sensor switching.

[0041] In one specific implementation, the surround-view warning domain controller receives radar calibration data of the target object from the millimeter-wave radar and unifies it into the vehicle coordinate system. Within the overlapping area, the detection rate of a single fisheye camera is 98.83%, the detection rate of a single narrow-angle camera is 92.86%, and the detection rate of a single millimeter-wave radar is 97.86%. The presence probability is fused by combining the detection rate, false alarm rate, target confidence, and existence period of the camera in this area, so that the fused detection rate reaches 99.26%. At the same time, the weighting coefficient is determined according to the detection accuracy variance of each sensor in the overlapping area and error correction is performed. The position and speed information are fused by weighted average. Furthermore, the vehicle speed and wheel angle are combined, and the sliding window method is used to determine whether the target object is stationary or moving. Finally, the fused target object status information containing the precise distance and motion status is output.

[0042] Step S50: Based on the vehicle status information, the panoramic surround view image, and the target object status information, conflict arbitration is performed through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller to determine the target regulatory warning task with the highest priority at present, and the currently triggered alarm signal and view switching command are output to the vehicle display device.

[0043] It should be noted that, in this embodiment, vehicle status information refers to data reflecting the current operating condition of the vehicle, including vehicle speed, gear (forward, reverse, neutral, etc.), turn signal status, steering wheel angle, etc.; blind spot unified warning rule refers to the hierarchical alarm and view linkage logic integrated in a single controller, which simultaneously covers and meets the requirements of relevant national regulations for lateral blind spots, rear blind spots, and forward blind spots; alarm signal refers to the audible and visual prompts output by the system according to the warning rules to alert the driver; view switching command refers to the control command used to control the in-vehicle display device to automatically switch between different screen views (such as surround view, forward, right side, and rear view); in-vehicle display device refers to the device on the vehicle used to display images and alarm information, including the vehicle's central control screen and the vehicle's instrument panel. This embodiment outputs alarms and view switching commands based on various information and warning rules. It aims to coordinate the warning needs of the three major regulatory blind spots through a unified set of warning rules, avoiding alarm conflicts and display jumps caused by multiple independent regulatory warning controllers in traditional solutions. It realizes graded alarms based on the degree of danger of the target and automatic view linkage. While meeting the mandatory warning requirements of regulations, it greatly improves the intelligence of human-computer interaction and driving safety.

[0044] Additionally, it should be noted that in one possible implementation, the blind spot unified warning rule adopts a two-level alarm mechanism: a normal alarm is a level one alarm, and a danger alarm is a level two alarm. This is achieved through visual color differentiation, icon differentiation, tiered alarm sounds, and instrument panel icons, creating a triple warning linkage, coupled with automatic perspective switching that adapts to driving conditions. For example, when the vehicle is in drive and the system triggers the side blind spot warning, the surround view warning domain controller outputs the corresponding alarm signal to the instrument panel and central control screen to display the corresponding icon. Simultaneously, it outputs a perspective switching command to display the surround view, front, and right-side views on the central control screen. If there is a right turn intention and a moving VRU target on the right, a level two alarm signal is output to control the flashing of the warning lights and the beeping alarm. When the vehicle is in reverse, a perspective switching command is output to switch the central control screen to the rear view. If a rear warning is triggered, the corresponding alarm icon and alarm sound are displayed.

[0045] This application uses multiple wide-angle cameras to collect surrounding images covering regulatory blind spots. The surround-view warning domain controller performs distortion correction and surround-view stitching based on chassis height adaptively switching calibration parameters. It also combines AI target recognition and radar data for multi-sensor fusion decision-making. Finally, based on vehicle status and preset rules, it outputs alarm signals and view switching commands in a unified manner. This solves problems such as difficulty in covering blind spots of special vehicles, stitching misalignment caused by chassis height changes, and conflicts in the integration of multiple controllers. It achieves 360° vehicle body monitoring without blind spots and high-precision ranging, significantly improving the VRU detection rate and reducing the false alarm rate. At the same time, the single controller's integration of multiple regulations significantly reduces hardware costs and certification cycle. Combined with automatic view switching and hierarchical alarm linkage mechanisms, it effectively improves driving and operation safety.

[0046] Furthermore, the acquisition of images around the vehicle via multiple wide-angle cameras includes: The vehicle collects forward and side-front image data using a first wide-angle camera and a second wide-angle camera installed on both sides of the front of the vehicle. Rear and side rear image data are collected by the third and fourth wide-angle cameras installed on both sides of the rear of the vehicle. High-resolution lateral image data is acquired by fisheye and narrow-angle cameras installed on the left and right sides of the vehicle, and lateral ranging data is acquired by millimeter-wave radar.

[0047] It should be noted that the first and second wide-angle cameras refer to the fisheye cameras installed on both sides of the front of the special vehicle crane. The forward and side-front image data refers to the visual information from the front and side front of the vehicle. To address the issue of blind spots caused by the protruding hook on the crane front, which is easily obstructed by a conventional single camera, the dual-camera arrangement on both sides of the front effectively avoids blind spots caused by the hook obstruction, achieving full coverage of blind spots and detection redundancy, ensuring no blind spots in the forward blind spot, and preventing the driver from missing any monitoring. The third and fourth wide-angle cameras refer to the fisheye cameras installed on both sides of the rear of the special vehicle crane. The rearward and side-rear image data refers to the visual information from the rear and side rear of the vehicle, covering the rearward regulatory blind spot and part of the side blind spot. To address the issue of single cameras being easily obstructed by spare tires or toolboxes typically located at the rear of cranes, a dual-camera setup on both sides of the rear effectively avoids blind spots caused by toolbox obstruction, achieving complete rear view coverage. This ensures blind-spot-free monitoring in scenarios such as low-speed parking and other situations requiring rear blind spots, compensating for blind spots near the vehicle body of large vehicles. Fisheye and narrow-angle cameras refer to combined visual sensors that include a fisheye lens with an ultra-wide field of view and a narrow-angle lens with a smaller field of view but high image resolution. High-resolution lateral image data refers to detailed image information at a greater distance to the side of the vehicle. Millimeter-wave radar refers to a sensor that uses millimeter-wave electromagnetic waves for target detection. Lateral ranging data refers to the distance and speed information of targets to the side of the vehicle. This embodiment utilizes a fisheye lens, a narrow-angle camera, and millimeter-wave radar to collaboratively acquire data. The aim is to achieve dual redundancy detection of lateral blind spots as required by relevant national regulations, combining visual and radar capabilities. The fisheye and narrow-angle cameras leverage their ultra-wide field of view and high resolution to achieve both close-range wide-field coverage and long-range target feature extraction. The millimeter-wave radar compensates for the inaccuracy of visual sensors in ranging and speed measurement under adverse weather conditions. Together, they acquire high-resolution lateral image data and lateral ranging data, providing accurate lateral target position and speed information for multi-sensor fusion decision-making. This achieves highly reliable lateral blind spot warnings and sensor degradation redundancy. The narrow-angle camera can cover vulnerable road users such as pedestrians and two-wheeled vehicles within a range of at least 30 meters behind the vehicle, providing high-resolution visual confirmation. The millimeter-wave radar supplements this with ranging and speed data at longer lateral distances, covering vehicle detection within a range exceeding 60 meters.

[0048] In addition, it should be noted that the fusion strategy of vision and radar can complement each other's strengths and weaknesses, enhance the detection capability of side blind spot warning, meet regulatory requirements when driving at high speeds, strengthen detection in accident-prone areas such as "inner wheel difference", and balance compliance and cost control.

[0049] Furthermore, referring to Figure 2 The second embodiment of the vehicle surround view warning method of this application provides a flowchart, based on the above. Figure 2 The embodiment shown further refines the step S20, which involves "adaptively switching the current calibration parameters based on the current chassis height information, and performing distortion correction and surround view stitching on the images surrounding the vehicle according to the current calibration parameters to generate a panoramic surround view image," and includes steps A201 to A204: Step A201: Based on the current chassis height information, the current homography matrix, current distortion correction coefficients, and current fusion weight table are obtained by matching from the pre-stored calibration parameter groups corresponding to different chassis heights.

[0050] It should be noted that in this embodiment, the current chassis height information refers to real-time data reflecting the current ground clearance of the vehicle chassis, which is usually obtained from feedback by the vehicle controller; the pre-stored calibration parameter sets corresponding to different chassis heights refer to multiple sets of calibration data that have been calibrated by the system under different chassis height conditions and stored in the system database according to height classification; the current homography matrix refers to the transformation matrix used to convert the image from the camera's view plane to the vehicle's top-down view plane; the current distortion correction coefficients refer to the intrinsic correction parameters used to eliminate optical distortion of the wide-angle camera; the current fusion weight table refers to the data table used to define the fusion ratio of each pixel in the overlapping area of ​​adjacent camera images; the surround view warning domain controller, based on the current chassis height information, matches the current homography matrix, current distortion correction coefficients, and current fusion weight table from the pre-stored calibration parameter sets corresponding to different chassis heights, aiming to achieve adaptive switching of calibration parameters for the variable chassis height of special vehicle cranes, thereby ensuring that the surround view stitching is not misaligned when the height changes, the target detection and ranging are accurate, and the warning function is not affected.

[0051] In one possible implementation, the pre-stored calibration parameter set corresponding to different chassis heights can not only include discrete parameters for specific height points, but also perform interpolation calculations based on parameters of adjacent height points to adapt to continuously changing chassis heights.

[0052] For example, in one specific implementation, when the crane adjusts the chassis height, the surround view warning domain controller receives the current chassis height information sent by the vehicle controller, then searches the local database for the pre-stored calibration parameter group that is closest to the height, and extracts the corresponding current homography matrix, current distortion correction coefficient and current fusion weight table.

[0053] Step A202: Based on the current distortion correction coefficients, perform distortion correction processing on the image around the vehicle and output the distortion-corrected image.

[0054] It should be noted that in this embodiment, the current distortion correction coefficient refers to the intrinsic correction parameters matched with the current chassis height and camera hardware, including radial and tangential distortion coefficients; the vehicle surrounding image refers to the original image with optical distortion captured by multiple wide-angle cameras; distortion correction processing refers to the algorithm operation of straightening and restoring curved edge lines in the original image using mathematical models and correction parameters; the distortion-corrected image refers to the image that conforms to the real physical perspective relationship after distortion correction. The surround view warning domain controller performs distortion correction processing on the vehicle surrounding image according to the current distortion correction coefficient and outputs a distortion-corrected image, aiming to overcome the shortcomings of fisheye cameras, which have a wide field of view but severe edge distortion, restore the real visual scene, and avoid target object position deviation caused by image distortion.

[0055] In one possible implementation, in order to reduce the computation time of the algorithm, the distortion correction process can retain the information of the effective region through the algorithm. By filtering the effective region, only the effective region is used for table lookup distortion correction, thereby reducing the redundant calculation caused by full pixel shift.

[0056] Step A203: Perform perspective transformation on the distorted image based on the current homography matrix to generate a bird's-eye view image.

[0057] It should be noted that in this embodiment, perspective transformation refers to the process of mapping an image from a tilted camera viewpoint to a top-down, vertically downward-facing viewpoint using matrix transformation; the bird's-eye view refers to the image of the ground viewed from directly above the vehicle after perspective transformation. The surround-view warning domain controller performs perspective transformation on the distortion-free image based on the current homography matrix to generate the bird's-eye view. This aims to unify the camera images from different perspectives into a single top-down coordinate system centered on the vehicle, providing a unified coordinate reference for seamless stitching of multiple images, ensuring consistent aspect ratios in the stitched image and preventing misalignment.

[0058] In one possible implementation, perspective transformation can be achieved through an inverse mapping interpolation algorithm to ensure that the generated bird's-eye view image is pixel-smooth and free of jagged edges. In a specific implementation, the surround view module uses the matched current homography matrix to perform perspective transformation on the distortion-free image generated by the forward wide-angle camera, and projects it onto a world coordinate system with the vehicle center as the origin to generate a bird's-eye view image of the forward region.

[0059] Step A204: According to the current fusion weight table, perform pixel-weighted fusion on the overlapping areas in the bird's-eye view images corresponding to adjacent wide-angle cameras to output a panoramic view image.

[0060] It should be noted that in this embodiment, the current fusion weight table refers to a parameter table defining the contribution ratio of each pixel in the overlapping area of ​​adjacent camera images from the left and right cameras respectively; adjacent wide-angle cameras refer to two adjacent cameras with partially overlapping fields of view; overlapping area refers to the area covered by the intersecting fields of view of two adjacent cameras; pixel weighted fusion refers to the processing method of calculating the final single pixel value by weighted averaging of multiple pixel values ​​at the same position in the overlapping area according to the weight table; panoramic surround view image refers to the 360-degree complete overhead view of the vehicle's surroundings formed by seamlessly stitching together all adjacent bird's-eye view images. The surround view warning domain controller performs pixel weighted fusion on the overlapping areas in the bird's-eye view images corresponding to adjacent wide-angle cameras according to the current fusion weight table, and outputs a panoramic surround view image. This aims to eliminate ghosting or seams caused by repeated images in the overlapping areas, making the transition between adjacent images smoother, realizing 360-degree all-round surround warning, providing the driver with a seamless panoramic view, and solving the target misalignment and disappearance phenomenon that easily occurs at the surround view stitching of large vehicles.

[0061] In one possible implementation, the fusion weights can be dynamically set based on the distance from the pixel to the center line of the overlapping area, with the weights transitioning more smoothly the closer they are to the center line.

[0062] The surround-view warning domain controller of this application adaptively matches the current homography matrix, distortion correction coefficients, and fusion weight table based on the current chassis height information. It then sequentially performs distortion correction, perspective change correction, and weighted fusion of overlapping pixels on the images around the vehicle to finally generate a panoramic surround-view image. This solves the problems of surround-view stitching misalignment, inaccurate ranging, and warning failure caused by chassis height changes in special vehicles such as cranes, and achieves dynamic adaptive switching of calibration parameters. At the same time, it overcomes the optical distortion defects of fisheye lenses and eliminates seams and ghosting in overlapping areas of multiple images through weighted fusion, achieving a smooth, seamless, and high-precision 360° panoramic surround-view display. This effectively avoids the misalignment and disappearance of targets in the blind spots near the vehicle body of large vehicles, improving driving and operational safety.

[0063] Furthermore, referring to Figure 3 The third embodiment of the vehicle surround view warning method of this application provides a flowchart, based on the above. Figure 3 The embodiment shown further refines the step S30, "using the pre-trained AI target detection model in the single surround-view warning domain controller to identify targets in the image around the vehicle," including steps A301 to A304: Step A301: Input the image around the vehicle into the pre-trained AI object detection model, and output the initial detection box set and the location confidence and classification confidence of each initial detection box.

[0064] It should be noted that, in this embodiment, the pre-trained AI object detection model refers to a deep learning model that has been trained offline with a large amount of labeled data and deployed on hardware, capable of extracting features and locating specific targets in an image. The initial detection box set refers to the set of two-dimensional rectangular boxes output by the model's initial inference to define the locations of all suspected targets in the image; the location confidence refers to the probability that the model judges the existence of an actual object within the initial detection box; the classification confidence refers to the probability that the model judges the object within the initial detection box to belong to a specific category (such as pedestrians, two-wheeled vehicles, etc.); the system inputs the image around the vehicle into the pre-trained AI object detection model and outputs the initial detection box set and the location confidence and classification confidence of each initial detection box, aiming to quickly extract all possible targets and their categories in the image through deep learning algorithms, thereby achieving preliminary identification of vulnerable road users (VRUs).

[0065] In one possible implementation, the pre-trained AI object detection model can be any model capable of performing image recognition and parsing tasks, such as a machine learning model, a neural network model, or a graph neural network model.

[0066] Step A302: Select the initial detection box with the highest location confidence as the reference box, calculate the intersection-union ratio (IU) of the remaining initial detection boxes with the reference box, determine the weight factor of the remaining initial detection boxes based on the IU, and perform weighted fusion of the initial detection boxes to obtain a unique detection box.

[0067] It should be noted that in this embodiment, the reference box refers to the initial detection box with the highest localization confidence among multiple overlapping initial detection boxes corresponding to the same target; the intersection-union ratio (IU) is the ratio of the intersection area to the union area of ​​two detection boxes, used to measure the degree of overlap between the two boxes; the weight factor is a coefficient determined based on the IU and used to adjust the contribution of different initial detection boxes in the fusion process; weighted fusion refers to the calculation process of weighted summation of the coordinates and size parameters of each initial detection box according to the weight factor; and the unique detection box refers to the single detection box obtained after weighted fusion of multiple overlapping initial detection boxes for the same target. In this embodiment, the initial detection box with the highest localization confidence is selected as the reference box, the IU of the remaining initial detection boxes with the reference box is calculated, and the weight factor of the remaining initial detection boxes is determined based on the IU. The initial detection boxes are then weighted and fused to obtain the unique detection box. The weighted non-maximum suppression (NMS) algorithm can be used to filter and merge overlapping bounding boxes of the same target to avoid duplicate counting. At the same time, compared with directly removing low-confidence boxes, weighted fusion can retain more boundary information, making the localization more accurate.

[0068] In one possible implementation, the larger the intersection-union ratio, the closer the remaining initial detection boxes are to the reference boxes, and the corresponding weight factors are set higher to give full play to the role of high-quality detection boxes.

[0069] Step A303: Multiply the location confidence of the unique detection box by the classification confidence to obtain the comprehensive confidence, and determine the unique detection box with the comprehensive confidence higher than the preset confidence threshold as a valid detection box.

[0070] It should be noted that, in this embodiment, the comprehensive confidence score refers to the joint probability value obtained by multiplying the location confidence score of the unique detection box by the classification confidence score, which is used to comprehensively assess the probability of the existence of a real target of a specific category within the detection box; the preset confidence threshold refers to a pre-set probability threshold used to distinguish between real targets and false positive targets; a valid detection box refers to a unique detection box whose comprehensive confidence score reaches or exceeds the preset confidence threshold, and is considered a real target. This embodiment obtains the comprehensive confidence score by multiplying the location confidence score of the unique detection box by the classification confidence score, and determines the unique detection box with a comprehensive confidence score higher than the preset confidence threshold as a valid detection box. The aim is to further filter out false detections caused by environmental interference by comprehensively evaluating the probability of target existence and the probability of category, thereby reducing the false alarm rate of the AI ​​algorithm and meeting the mandatory requirements of regulations for target detection accuracy.

[0071] In one possible implementation, the preset confidence threshold is not a fixed value, but rather different confidence thresholds are set according to different obstacle types. For example, different threshold parameters are set for false alarms from trash cans at night and false alarms from water accumulation on the ground, thereby accurately filtering false alarms in various specific scenarios.

[0072] Step A304: Match the valid detection boxes with the target list of the current tracking algorithm. For the successfully matched valid detection boxes, use the Kalman filter algorithm to update the state and output smooth detection boxes. Add the unmatched valid detection boxes as new targets to the target list of the tracking algorithm. Remove the unmatched targets from the list for a consecutive preset period to output the finally identified target.

[0073] It should be noted that, in this embodiment, the current tracking algorithm target list refers to the set of target detection boxes that the system has recorded in historical frames and are continuously tracking; the Kalman filter algorithm refers to an algorithm that uses the linear system state equation to make the optimal estimate of the system state through system input and output observation data; state update refers to the process of correcting the predicted state of existing targets in the current tracking algorithm target list by combining the observation data of the current frame; smooth detection box refers to a detection box whose position and size changes are more continuous and stable and less prone to jitter after Kalman filtering; new target refers to a valid detection box that appears in the current frame but cannot be found in the current tracking algorithm target list; preset period refers to the maximum number of consecutive frames or time period that the system allows for target loss; and the finally identified target refers to the stable and real target object output to the display screen after tracking filtering and elimination logic. This embodiment matches valid detection boxes with the target list of the current tracking algorithm. For successfully matched valid detection boxes, a Kalman filter algorithm is used to update the state and output smooth detection boxes. Valid detection boxes that fail to match are added as new targets to the target list of the current tracking algorithm. Targets that fail to match for a consecutive preset period are removed from the list to output the final identified target. Tracking filtering is performed on each valid detection box to make the output detection box more stable and smooth, avoiding flickering and jumping of the detection box. At the same time, false alarms that occur briefly and lost targets that have gone out of the detection range are removed, which greatly improves the accuracy and robustness of the algorithm.

[0074] This application inputs images of the vehicle's surroundings into a pre-trained AI target detection model to obtain initial detection boxes and confidence scores. After weighted fusion using intersection-union-ratio (IU) to remove duplicates and filtering using a comprehensive confidence score threshold, the model combines the target list from the tracking algorithm with Kalman filtering for state updates and target addition / deletion management, outputting the final identified target. This embodiment avoids duplicate counting of the same target through weighted fusion, effectively suppresses false alarms in complex scenarios by combining comprehensive confidence scores and tracking cycle thresholds, and ensures smooth and stable target tracking using Kalman filtering, significantly improving the accuracy and robustness of AI detection and meeting the mandatory requirements of regulations for accurate identification of vulnerable road users (VRUs) and low false alarm rates.

[0075] Furthermore, the step of calculating the visual target position information of the target object through pixel fitting and region segmentation algorithms includes A401~A404: Step A401: Based on the pixel position of the target object in the image around the vehicle, the area where the target object is located is divided into regions, and the divided regions are filtered for effective regions.

[0076] It should be noted that in this embodiment, the target object refers to objects such as pedestrians, bicycles, and vehicles that need to be monitored in the environment surrounding the vehicle; region division refers to the process of dividing the image into different regions based on the imaging characteristics of the camera and the differences in ranging accuracy; effective region filtering refers to the process of retaining information on reliable ranging regions through algorithms, eliminating regions with large errors such as those near the vanishing point, and only performing table lookup calculations on the effective regions. This embodiment divides the area where the target object is located based on the pixel position of the target object in the image around the vehicle, and performs effective region filtering on the divided regions. This overcomes the ranging error problem caused by the defects of fisheye cameras and the lack of depth information. Especially when the error is greater closer to the vanishing point, by retaining reliable dimensions of each camera through filtering, the computation time of the algorithm is greatly reduced, and the real-time performance of the system is improved.

[0077] In one possible implementation, the effective region can be filtered based on the distance between the pixel's ordinate and the vanishing point. That is, based on the distance of the target object's pixel position from the image's vanishing point, after dividing the region, a threshold line for the effective ordinate is set for filtering. The effective region can also be filtered based on the degree of image edge distortion. Since there is severe barrel distortion on both sides of the horizontal edge of the image around the vehicle, the ratio coefficient between the pixel coordinates of the edge region and the world coordinates is no longer linearly related. If the edge pixels are forcibly used for distance measurement, the calculated visual coordinate information will have a large deviation. After dividing the region, an edge band of a certain pixel width is defined on the left and right sides of the image as an invalid region. When the pixel position of the target object falls within the edge band, it is filtered out. Only the central and near-central regions with less distortion and high calibration accuracy in the image are retained as effective regions, ensuring that the data involved in the pixel difference calculation all come from high-confidence regions.

[0078] Step A402: Within the selected valid area, obtain the pre-calibrated ratio coefficient between world coordinates and pixel coordinates.

[0079] It should be noted that, in this embodiment, the ratio of world coordinates to pixel coordinates refers to the conversion ratio between the actual physical space distance and the image pixel distance.

[0080] Step A403: Calculate the pixel difference between the pixel coordinates of the target object and the reference pixel coordinates based on the scaling factor.

[0081] It should be noted that, in this embodiment, pixel difference refers to the difference between the current pixel coordinates of the target object and the reference pixel coordinates.

[0082] Step A404: Calculate the visual target position information of the target object in the vehicle coordinate system based on the pixel difference and the scaling factor using a pixel fitting algorithm.

[0083] It should be noted that, in this embodiment, the pixel fitting algorithm refers to the calculation method that maps the offset of the image pixel plane into real physical coordinates through a mathematical geometric model; the visual target position information refers to the precise physical distance and orientation information of the target object in the vehicle coordinate system.

[0084] This embodiment obtains a pre-calibrated ratio coefficient between world coordinates and pixel coordinates within the selected effective area, and calculates the pixel difference between the target's pixel coordinates and the reference pixel coordinates based on this ratio coefficient. Then, a pixel fitting algorithm is used to calculate the visual target position information of the target in the vehicle coordinate system based on the pixel difference and the ratio coefficient. The aim is to achieve accurate ranging using monocular vision, avoiding the inaccurate ranging problem caused by traditional full-pixel shift distortion processing, controlling the forward longitudinal ranging error to ≤15cm and the lateral error to ≤10cm, accurately matching the requirements for target positioning accuracy, and ensuring that the alarm distance judgment meets the regulatory threshold standards.

[0085] In one possible implementation, the effective area selection strategy and scaling factor are different for wide-angle cameras at different positions and angles. For example, in the fusion area of ​​two cameras, the lateral distance information of each camera is more reliable. Therefore, the lateral pixel difference is retained first for fitting calculation, thereby greatly improving the ranging accuracy of the fusion area.

[0086] In one possible implementation, the pixel fitting algorithm can employ a geometric fitting model based on pinhole imaging inverse projection, combining the product of pixel difference and scaling factor with the camera's pitch and yaw angles to perform trigonometric function calculations; alternatively, it can employ an offline calibrated polynomial fitting model, directly substituting the pixel difference as an independent variable into the polynomial equation to solve for the physical distance in the vehicle coordinate system.

[0087] Furthermore, referring to Figure 4 The fourth embodiment of the vehicle surround view warning method of this application provides a flowchart, based on the above. Figure 4 The embodiment shown further refines the step S40, "using the single surround-view warning domain controller to perform multi-sensor fusion decision-making on the radar target position information and the visual target position information, and outputting the fused target state information," including steps A501 to A506: Step A501: Receive the radar coordinate information collected by the radar sensor, and combine it with the installation position of the radar sensor to unify the radar coordinate information into the vehicle coordinate system to obtain the radar target position information.

[0088] It should be noted that, in this embodiment, radar sensors refer to radio wave or acoustic wave detection devices used to detect the position and velocity of targets, including millimeter-wave radar and ultrasonic radar; the installation position of the radar sensor refers to the physical fixed coordinates of the radar on the vehicle body; and the radar target position information refers to the spatial position data of the target object in the vehicle coordinate system. The single surround-view warning domain controller receives the radar coordinate information collected by the radar sensor, combines it with the installation position of the radar sensor, and unifies the radar coordinate information to the vehicle coordinate system to obtain the radar target position information. This aims to eliminate the inconsistency of spatial reference systems caused by the differences in the installation positions of different sensors, thereby ensuring the accuracy of multi-sensor data fusion.

[0089] In one possible implementation, the radar sensor types include millimeter-wave radar and ultrasonic radar. For different types of radar sensors, a coordinate-unified transformation matrix is ​​generated by calibration according to their specific installation offset and angular offset.

[0090] In one possible implementation, the coordinate unification process can be achieved through a homogeneous coordinate transformation matrix, which contains a translation vector determined based on the installation position of the radar sensor and a rotation matrix determined based on the radar orientation. Multiplying the radar coordinate information with this transformation matrix yields the radar target position information.

[0091] Step A502: Based on the visual target position information and the radar target position information, the same target objects are associated and matched, and the detection area type of the associated target object is determined. The detection area type includes single sensor detection area and multi-sensor detection overlap area.

[0092] It should be noted that in this embodiment, visual target position information refers to the position data of the target object in the vehicle coordinate system calculated by visual algorithms; radar target position information refers to the radar detection position data after coordinate unification; association matching refers to the process of binding data of the same physical target observed by different sensors; detection area type refers to the types of areas divided according to the coverage of the sensor's field of view; single sensor detection area refers to the area covered by only one sensor's field of view; multi-sensor detection overlapping area refers to the area where the fields of view of two or more sensors intersect and overlap. The fusion decision module associates and matches the same target objects based on visual coordinate information and radar target coordinate information, and determines the detection area type of the associated target object. The detection area type includes single sensor detection area and multi-sensor detection overlapping area. The aim is to accurately select the corresponding data processing strategy according to the position of the target object. For overlapping areas, a complex fusion algorithm is used to take advantage of each other's strengths and weaknesses, while for non-overlapping areas, single sensor data is used directly, thereby optimizing the system's computing power allocation while ensuring detection performance.

[0093] In one possible implementation, association matching can be performed by calculating the spatial Euclidean distance between visual coordinate information and radar target coordinate information, and then classifying targets with a distance less than a preset threshold as the same target and binding them together.

[0094] For example, in one specific implementation, the fusion decision module calculates the distance between the pedestrian visual coordinate information identified by the left fisheye camera and the radar target coordinate information of the left millimeter-wave radar. After confirming that the two point to the same pedestrian, it performs association matching and determines that the pedestrian is in the multi-sensor detection overlap area where the field of view of the left camera and the left millimeter-wave radar intersects based on the pedestrian's coordinate position.

[0095] Step A503: If the target object is located in the overlapping area of ​​multi-sensor detection, the existence probability of the target object is fused by combining the detection rate, false alarm rate, target confidence and existence period of each sensor in the overlapping area of ​​multi-sensor detection to obtain the fused existence probability.

[0096] It should be noted that in this embodiment, the detection rate refers to the probability that the sensor successfully detects a real target within a specific area; the false alarm rate refers to the probability that the sensor misidentifies a non-target object as a target object; the target confidence level refers to the probability value that the AI ​​model or radar judges the current detection frame to be a real target; the existence period refers to the number of frames or the length of time that the target object is continuously detected by the sensor; the existence probability fusion refers to the process of jointly reasoning and calculating the probability of the target's actual existence by combining parameters such as the detection rate and false alarm rate of multiple sensors; the fused existence probability refers to the final probability value of the target's actual existence obtained after the fusion of multi-sensor information. If the target object is located in the overlapping area of ​​multi-sensor detection, the fusion decision module combines the detection rate, false alarm rate, target confidence level, and existence period of each sensor in the overlapping area of ​​multi-sensor detection to perform existence probability fusion of the target object, obtaining the fused existence probability. Through multi-calculation and comprehensive reasoning, the complementarity between multiple sensors is used to reduce the false alarm rate caused by missed detections, and cross-validation is used to reduce the false alarm rate caused by false detections, thereby obtaining a more accurate target existence probability, making the system's identification of real targets more reliable.

[0097] In one possible implementation, probability fusion can employ a Bayesian inference algorithm, using the detection rate and false alarm rate of each sensor as prior probabilities, and combining them with the target confidence level to iteratively calculate the posterior probability.

[0098] For example, in one specific implementation, within the overlapping area detected by the first multi-sensor system, the system combines the 98.83% detection rate of the fisheye camera, the 97.86% detection rate of the millimeter-wave radar, as well as the target confidence and existence period, to perform a fusion calculation of the existence probability of the same pedestrian target, thereby increasing the fusion detection rate corresponding to the fusion existence probability to 99.26%, which significantly reduces the false alarm rate of a single sensor.

[0099] Step A504: Determine the weighting coefficients based on the detection accuracy variance of each sensor within the overlapping area of ​​the multi-sensor detection, and perform a weighted average of the target position and velocity information provided by each sensor based on the weighting coefficients to obtain the fused position and velocity information.

[0100] It should be noted that in this embodiment, the detection accuracy variance refers to the statistical measure of the deviation between the measured value and the true value when the sensor measures the same target position multiple times; the weighting coefficient refers to the proportion factor assigned to the data of each sensor in the fusion calculation; the target position and velocity information refers to the coordinate value and movement speed of the target object in the vehicle coordinate system; the weighted average refers to the mathematical operation of summing and averaging the measured values ​​of multiple sensors according to different weighting coefficients; the fused position and velocity information refers to the comprehensive position and velocity result obtained after weighted average calculation; the fusion decision module determines the weighting coefficient based on the detection accuracy variance of each sensor in the overlapping area of ​​multi-sensor detection, and performs a weighted average of the target position and velocity information provided by each sensor based on the weighting coefficient to obtain the fused position and velocity information. The aim is to dynamically allocate the confidence level according to the accuracy performance of the sensor in different areas, retain the reliability dimension of the high-precision sensor, so that the final fused position and velocity information is more accurate and more stable than the single sensor information, thereby meeting the strict requirements of regulations on ranging error.

[0101] In one possible implementation, the weighting coefficients are calculated not only based on the detection accuracy variance, but also by introducing errors to correct the weighting factors, in order to overcome the bias caused by the non-unbiased estimation of the sensor detection results.

[0102] For example, in one specific implementation, for targets in the overlapping area detected by multiple sensors, the system calculates the weight coefficients of each based on the characteristics of small lateral ranging variance of fisheye camera and small longitudinal ranging variance of millimeter-wave radar. It assigns higher weight to the lateral visual coordinates and higher weight to the longitudinal radar coordinates, and obtains the fused position and velocity information after weighted averaging, which significantly reduces the absolute error and the error fluctuation range is smaller.

[0103] Step A505: Combining the vehicle's speed and wheel angle, the actual speed of the target object is calculated by compensating the fused position and speed information. The target object is then judged to be in motion when the actual speed continuously exceeds the motion judgment threshold using the sliding window method.

[0104] It should be noted that in this embodiment, vehicle speed refers to the vehicle's current driving speed; wheel angle refers to the current deflection angle of the vehicle's steering wheel or wheels; compensation calculation refers to the process of correcting the relative speed of the target by combining the vehicle's own motion state to obtain the absolute motion speed; the actual speed of the target object refers to the actual motion rate of the target object in the global ground coordinate system; the sliding window method refers to a temporal logic judgment method that uses continuous data within a fixed time window to determine the state. The motion judgment threshold refers to the speed threshold used to distinguish whether the target object is in a stationary or moving state; the motion state refers to the dynamic attribute of the target object undergoing spatial displacement. The fusion decision module combines the vehicle's speed and wheel angle to perform compensation calculations on the fused position and speed information to obtain the actual speed of the target object, and uses the sliding window method to determine the target object as being in a moving state when the actual speed continuously exceeds the motion judgment threshold, accurately distinguishing between truly moving vulnerable road users (VRUs) and stationary obstacles, avoiding false alarms for stationary obstacles such as green belts and parked vehicles during driving, and optimizing the driver experience while meeting regulatory requirements.

[0105] In one possible implementation, the compensation calculation is achieved by decomposing the vehicle speed along the wheel rotation angle and then vector-adding it to the relative speed of the target, thereby eliminating the influence of the vehicle's own motion on the target speed observation.

[0106] In one possible real-time approach, the formula for calculating the actual velocity of the target object is: obj_vx = cos(wheel_angle (M_PI / 180.0)) vehicle_velocity_kph / 3.6 +relative_vx; obj_vy = sin(wheel_angle (M_PI / 180.0)) vehicle_velocity_kph / 3.6 +relative_vy; Where, obj_vx and obj_vy refer to the actual longitudinal and lateral velocities of the target object, respectively; vehicle_velocity_kph refers to the vehicle speed (in km / h); 3.6 is a speed unit conversion factor used to convert vehicle speed from km / h to m / s (1 m / s = 3.6 km / h); wheel_angle refers to the wheel angle (in degrees, °), representing the deflection angle of the current wheel relative to the longitudinal axis of the vehicle body; M_PI refers to the constant value of pi π (approximately 3.1415926); (M_PI / 180.0) refers to the conversion factor from angle to radians, which converts the wheel rotation angle from angle to the radian system required for trigonometric functions; relative_vx and relative_vy refer to the longitudinal relative velocity (along the X-axis of the vehicle coordinate system, in m / s) and lateral relative velocity (along the X-axis of the vehicle coordinate system, in m / s) of the target object relative to the vehicle, respectively; in this embodiment, the vehicle's own motion velocity is decomposed along the wheel rotation direction to the X and Y axes of the vehicle coordinate system, and then vector-superimposed with the relative velocity measured by the radar, thereby compensating for the influence of the vehicle's own motion on the target velocity observation, and finally obtaining the actual motion velocity of the target object in the ground coordinate system.

[0107] Step A506: Combine the fusion existence probability, fusion position and velocity information and the motion state into the fused target object state information.

[0108] It should be noted that in this embodiment, the fused existence probability refers to the final confidence level of the target's actual existence after multi-sensor probabilistic inference; the fused position and velocity information refers to the high-precision target spatial coordinates and motion rate after weighted averaging; and the fused target state information refers to a comprehensive data set including the target's existence probability, spatial position, motion velocity, and dynamic / static state. The fusion decision module combines the fused existence probability, fused position and velocity information, and motion state into the fused target state information. Its purpose is to integrate multi-dimensional, scattered attributes into a structured, complete target description, ensuring that the system can accurately judge and warn of blind zone risks in accordance with regulatory requirements.

[0109] In one possible implementation, the fused target status information is encapsulated in a specific data structure or CAN message format and synchronously output to the vehicle display screen to show the detected target with a visual tracking identifier.

[0110] This application unifies radar coordinates to the vehicle coordinate system and correlates them with visual information to determine the detection area. It then performs existence probability fusion and weighted position-velocity fusion on targets in overlapping multi-sensor areas. Combined with vehicle state compensation, it calculates the target's actual speed and uses a sliding window method to determine its motion state. Finally, it outputs the fused target state information. This embodiment achieves complementary advantages between visual and radar data through coordinate system one and multi-sensor correlation matching. Existence probability fusion significantly reduces the false alarm and false alarm rates, while weighted position-velocity fusion improves the accuracy and stability of target localization and speed measurement. The combination of vehicle motion compensation and the sliding window method for motion state determination effectively filters out false alarms about stationary obstacles.

[0111] Furthermore, referring to Figure 5 The fifth embodiment of the vehicle surround view warning method of this application provides a flowchart, based on the above. Figure 5 The embodiment shown further refines the step S40, which involves "receiving radar target position information detected by radar sensors, performing multi-sensor fusion decision-making on the radar target position information and the visual target position information through the single surround-view warning domain controller, and outputting the fused target state information," by including steps A601 to A604: Step A601: Real-time detection of the health status indicators of each sensor.

[0112] It should be noted that in this embodiment, "sensors" refers to various vehicle-mounted sensing hardware, including wide-angle cameras, narrow-angle cameras, millimeter-wave radar, and ultrasonic radar. Health status indicators are markers or status words used to characterize whether a sensor is currently functioning normally, whether the communication link is unobstructed, whether the lens is blocked, or whether a hardware malfunction has occurred. This embodiment monitors the health status indicators of each sensor in real time to promptly detect hardware failures or detection obstructions, providing accurate triggering conditions for the degradation redundancy mechanism. This ensures the system can respond quickly in the event of hardware anomalies, avoiding the complete loss of blind spot detection capability due to the failure of a single sensor.

[0113] In one possible implementation, the health status indicator includes not only hardware fault codes actively reported by the sensors, but also occlusion or failure indicators automatically diagnosed by the system based on data continuity, signal-to-noise ratio, or image characteristics.

[0114] In one specific implementation, the surround-view warning domain controller reads the health status indicators of the right-side millimeter-wave radar and the right-side narrow-angle camera in real time at a frequency of 10Hz through the communication parsing module, and determines whether their communication status is normal and whether the target data output is continuous.

[0115] Step A602: When any main sensor fails, determine the type of the failed sensor and the regulatory blind area it covers based on the health status identifier, and output the failed sensor information.

[0116] It should be noted that, in this embodiment, the main sensor refers to the sensor that undertakes the main detection task and has the highest detection accuracy or confidence level in a certain regulatory blind zone area; failure refers to the state in which the sensor cannot provide effective detection data due to hardware failure, communication interruption, severe lens obstruction, or severe data distortion; the type of failed sensor refers to the specific category of the failed sensor, such as a wide-angle camera or millimeter-wave radar; the covered regulatory blind zone area refers to the physical space range of the corresponding regulatory requirements that the failed sensor was originally responsible for detecting, such as a lateral blind zone, a rear blind zone, or a forward blind zone. Failed sensor information refers to comprehensive descriptive data that includes attributes such as the type of failed sensor, the corresponding regulatory blind zone area, and the time of failure. When any main sensor failure is detected, the system determines the type of failed sensor and the covered regulatory blind zone area based on the health status identifier, outputs the failed sensor information, accurately locates the source of the failure and the scope of the affected regulatory blind zone, and clarifies which regulatory blind zone areas have lost their main detection support, so as to initiate a redundancy completion strategy accordingly.

[0117] In one possible implementation, the roles of the main sensor and the auxiliary sensor can be dynamically adjusted according to different vehicle operating conditions. For example, at low speeds, the main sensor is a wide-angle camera, and at high speeds, the main sensor is a millimeter-wave radar.

[0118] Step A603: Based on the information of the failed sensor, reset the weight of the failed sensor to zero, and redistribute the weight of the failed sensor to the remaining available sensors covering the same regulatory blind zone area according to the preset confidence level allocation rule, and output the adjusted fusion weight set.

[0119] It should be noted that in this embodiment, the failed sensor information is data describing the attributes and impact range of the failed sensor; resetting the weights to zero refers to forcibly setting the weight coefficients corresponding to the failed sensor to 0 in multi-sensor fusion calculations to completely eliminate the interference of its distorted data on the fusion results; the preset confidence allocation rule refers to the pre-set logic and strategy for transferring the weights of the main sensor to other auxiliary sensors in overlapping areas when the main sensor fails; the remaining available sensors refer to auxiliary or redundant sensors covering the same regulatory blind area and whose current health status is marked as normal; the adjusted fusion weight set refers to a new set of weight coefficients used to guide the current fusion calculation after the weights have been redistributed. This embodiment, based on the failed sensor information, resets the weights of the failed sensor to zero and redistributes the weights of the failed sensor to the remaining available sensors covering the same regulatory blind area according to the preset confidence allocation rule, outputting the adjusted fusion weight set. The aim is to dynamically adjust the fusion architecture, utilize redundant sensors to take over the detection weights of the failed sensor, ensure that the core regulatory early warning function is not interrupted, avoid system-wide paralysis caused by a single sensor failure, and achieve safety redundancy protection of "main sensor + auxiliary sensor".

[0120] In one possible implementation, the preset confidence allocation rule can be to transfer all the weights of the failed sensor to the single remaining usable sensor with the closest detection accuracy, or it can be to distribute the weights of the remaining usable sensors proportionally to the variance of their detection accuracy.

[0121] In one specific implementation, a preset confidence allocation rule dynamically allocates confidence scores based on the historical statistical performance of each remaining available sensor in the overlapping area. Specifically, the system pre-stores the detection rate (PD) and false alarm rate (FAR) of each sensor in different regulatory blind zones (such as lateral blind zones). When the primary sensor (such as a wide-angle camera) fails, the system extracts the detection rate and false alarm rate of the remaining available sensors (such as narrow-angle cameras and millimeter-wave radars) covering the same regulatory blind zone and calculates their respective "confidence scores" (e.g., score = PD / (1 + FAR)). Subsequently, the confidence scores of each remaining available sensor are normalized and used as weighting coefficients for redistribution.

[0122] In one specific implementation, considering the significant differences in ranging characteristics between visual sensors and radar sensors, the confidence allocation rule employs a spatial dimension decoupling approach for weight redistribution. Specifically, independent weight allocation strategies are set for the target's lateral position information (Y-axis) and longitudinal position information (X-axis). When the main sensor (wide-angle camera) fails, its failure causes a simultaneous decrease in ranging accuracy in both the lateral and longitudinal dimensions. The system then splits its original weights: in the lateral position dimension, because the narrow-angle camera exhibits less distortion and lower ranging variance in the lateral edge region, the system allocates 80% of the main sensor's lateral weights to the narrow-angle camera and only 20% to the millimeter-wave radar; in the longitudinal position dimension, because the millimeter-wave radar has high ranging accuracy and is unaffected by lighting conditions, the system allocates 80% of the main sensor's longitudinal weights to the millimeter-wave radar and only 20% to the narrow-angle camera. This decoupled allocation maximizes the advantages of the remaining available sensors in their respective strengths.

[0123] Step A604: Based on the adjusted fusion weight set, perform existence probability fusion on the detection data of the remaining available sensors, and output the downgraded fusion target state information.

[0124] It should be noted that in this embodiment, the remaining available sensors refer to auxiliary sensors that are still working normally within the same regulatory blind zone; existence probability fusion refers to the process of jointly reasoning and calculating the true existence probability of a target by combining information such as the detection rate, false alarm rate, target confidence, and existence period of each available sensor; and the downgraded fused target state information refers to the comprehensive state data such as target position, velocity, and existence probability calculated solely from the remaining available sensors when some sensor data is lost. This embodiment performs existence probability fusion on the detection data of the remaining available sensors based on the adjusted fusion weight set, and outputs downgraded fused target state information. This aims to ensure that even under severe conditions of sensor failure, the blind zone detection capability required by regulations can still be supplemented through the fusion of data from the remaining available sensors, ensuring that the core regulatory early warning function remains uninterrupted.

[0125] In one possible implementation, the downgraded fusion target state information may include a downgrade flag, which is used to alert the driver on the vehicle display that the current perception system is not fully operational.

[0126] This application addresses the problem of blind zone warning function interruption caused by single sensor failure by real-time detection of sensor health status and determination of its type and coverage blind zone when the main sensor fails. It resets the weights of the failed sensor to zero and redistributes them to the remaining available sensors in the same blind zone according to preset rules. Based on the adjusted weight set, it performs existence probability fusion on the remaining data to output the downgraded fused target status information. This solves the problem of blind zone warning function interruption caused by single sensor failure. By dynamically redistributing weights and fusing redundant sensor data to supplement detection capabilities, it ensures the continuous effectiveness of the core regulatory warning function and significantly improves the system's safety redundancy, reliability and fault tolerance.

[0127] Furthermore, the steps of determining the highest priority target regulatory warning task based on vehicle status information, the panoramic surround view image, and the target object status information, through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller, and outputting the currently triggered alarm signal and view switching command to the vehicle display device include A701~A704: Step A701: Based on the fused position and speed information and motion state in the target object state information, and combined with the gear position, vehicle speed and turn signal status in the vehicle state information, the warning trigger conditions of the first target regulation, the second target regulation and the third target regulation are evaluated in parallel, and corresponding regulation warning tasks are generated.

[0128] It should be noted that: fused position refers to the precise spatial coordinates and distance data of the target object in the vehicle coordinate system; motion state refers to the dynamic attribute of the target object after being judged by the sliding window method, whether it is stationary or in motion; vehicle status information refers to the set of parameters reflecting the current operating condition of the vehicle; parallel evaluation refers to the calculation process of a single surround-view warning domain controller simultaneously executing multiple regulatory judgment logics within the same processing cycle; warning trigger conditions refer to the risk judgment standards defined by each target regulation regarding the distance of the target object, collision time, and vehicle operating behavior; regulatory warning task refers to the set of instructions generated when the target object status and vehicle status meet the warning trigger conditions of a specific regulation, used to drive subsequent alarm and display operations.

[0129] This embodiment aims to fully leverage the centralized processing advantages of a single computing platform to comprehensively assess the regulatory risks of vehicles in the forward, backward, and lateral directions simultaneously. This avoids the decision delays caused by information silos in traditional distributed controllers, thereby generating comprehensive regulatory warning tasks in the shortest possible time and providing a complete pool of candidate instructions for subsequent unified arbitration.

[0130] In one possible implementation, the parallel evaluation process can be based on a multi-threaded architecture. Within a single surround-view warning domain controller, independent evaluation threads are allocated for the first target regulation, the second target regulation, and the third target regulation. Each thread simultaneously reads the same frame of target status information and vehicle status information and makes independent logical judgments.

[0131] For example, if a single surround-view warning domain controller reads the target status information within the same clock cycle, showing that there is a bicycle 3 meters to the right, and the vehicle status information shows that the current gear is forward, the speed is 15 km / h, and the right turn signal is activated, the single surround-view warning domain controller will simultaneously evaluate the warning trigger conditions of the first target regulation, the second target regulation, and the third target regulation. If it finds that the side blind spot warning requirements of the third target regulation are met, it will then generate the corresponding side warning task for the third target regulation.

[0132] Step A702: When the number of regulatory warning tasks triggered at the same time is greater than or equal to two, the cross-regulatory priority arbitration mechanism is activated to arbitrate the conflict of the simultaneously triggered regulatory warning tasks according to the preset regulatory priority sequence; the regulatory priority sequence is set such that the priority of the backward warning task corresponding to the second target regulation is greater than the priority of the lateral warning task corresponding to the third target regulation, and the priority of the lateral warning task corresponding to the third target regulation is greater than the priority of the forward warning task corresponding to the first target regulation.

[0133] It should be noted that the cross-regulatory priority arbitration mechanism refers to the logical judgment rule in which a single look-around warning domain controller selects and retains the only highest priority task according to the preset priority order when multiple regulatory warning tasks are triggered at the same time; conflict arbitration refers to the decision-making process in which the system selects the best task and blocks other low-priority tasks when there are multiple competing regulatory warning tasks at the same time.

[0134] This embodiment aims to solve the problem of multiple regulatory alarms sounding simultaneously in complex scenarios (such as when a vehicle is coasting in gear and reversing into a parking space with the turn signal on), which can cause driver panic and distraction. Through strict sequence arbitration, it ensures that the system outputs only the single regulatory warning task that best matches the current high-risk scenario at any given time, thereby greatly reducing alarm noise interference in the cockpit and improving the driver's focus on the most deadly dangers.

[0135] In one possible implementation, the preset regulatory priority sequence can be dynamically configured and updated according to the actual usage scenarios of different vehicle models or the level of regulatory enforcement in different countries and regions. For example, for specific vehicle models such as buses, the priority of the forward warning task corresponding to the first target regulation can be raised to the highest level.

[0136] Step A703: Determine the highest priority target regulatory warning task based on the conflict arbitration result, and output a graded alarm signal according to the danger level threshold corresponding to the target regulatory warning task.

[0137] It should be noted that the conflict arbitration result refers to the final judgment state that determines which regulatory warning task is granted the right to be executed after being screened by the cross-regulatory priority arbitration mechanism; the graded alarm signal refers to the warning instructions with different sound and light intensities output by a single surround-view warning domain controller according to the degree of danger of the target object.

[0138] This embodiment aims to determine the most critical early warning task and then further quantify and classify the urgency of the danger of the target object. This avoids the problem that a single, crude alarm method cannot allow the driver to accurately perceive the level of risk. Through the refined judgment of multi-level thresholds, different frequencies of beeping sounds or different colors of screen flashing signals are output, so that the driver can intuitively and accurately grasp the current level of danger and take corresponding braking or evasive measures.

[0139] In one possible implementation, the danger level threshold can be divided into three levels: when the distance to the target is less than the first distance threshold, a level one alert alarm signal is output; when it is less than the second distance threshold, a level two warning alarm signal is output; and when it is less than the third distance threshold, a level three emergency alarm signal is output.

[0140] Step A704: Generate a perspective switching command that matches the highest priority target regulatory warning task. The perspective switching command has a higher priority than perspective switching commands triggered by normal gear position and turn signal status, so as to force the vehicle display device to switch to the forward view, rear view, side view or panoramic surround view image with the target highlighted corresponding to the graded alarm signal.

[0141] It should be noted that the perspective switching command refers to the control command that controls the vehicle display device to change the content of the currently displayed screen; the perspective switching command triggered by the normal gear position and turn signal status refers to the normal screen switching command generated based on the driver's normal operating intentions (such as shifting into reverse to look behind, or turning on the turn signal to look to the side); the panoramic view image with the target highlighted refers to the augmented reality image on the 360-degree bird's-eye view stitched image where dangerous targets are marked with a red border or flashing.

[0142] This embodiment aims to ensure that the presentation of the visual image is absolutely synchronized with the most dangerous alarm task. By using a forced overlay mechanism, it breaks the driver's possible incorrect perspective and focuses their attention, forcing their eyes to immediately focus on the location of the danger source that triggered the highest priority alarm. This achieves perfect linkage of sound, light and visuals in emergency situations, minimizing the driver's reaction time from hearing the alarm to seeing the dangerous target.

[0143] In one possible implementation, the forced switching logic of the view switching command can be set to keep the screen magnified for a predetermined duration (e.g., 3 seconds), during which other view switching requests manually triggered by the driver are blocked to ensure that the hazard is fully addressed.

[0144] For example, the single surround view warning domain controller generates a perspective switching command that matches the rearward warning task corresponding to the second target regulation. This command has a higher priority than the conventional forward view display command caused by the vehicle being in drive. It forces the vehicle display device to immediately switch from the central control main interface to a panoramic surround view image with a red highlight of the pedestrian behind, so that the driver can immediately lock onto the dangerous pedestrian behind on the screen when he hears the level 3 emergency alarm signal.

[0145] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the vehicle surround view warning method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0146] This application also provides a vehicle surround view warning device, please refer to... Figure 6 The vehicle surround view warning device includes: Image acquisition module 10 is used to acquire images around the vehicle through multiple wide-angle cameras, wherein the multiple wide-angle cameras include at least wide-angle cameras that cover the forward regulatory blind spot, the rear regulatory blind spot and the lateral regulatory blind spot; The image calibration module 20 is used to receive the current chassis height information of the vehicle through a single surround view warning domain controller, adaptively switch the current calibration parameters based on the current chassis height information, and perform distortion correction and surround view stitching on the images around the vehicle according to the current calibration parameters to generate a panoramic surround view image. The target detection module 30 is used to identify targets in the images around the vehicle using the pre-trained AI target detection model in the single surround view warning domain controller, and to calculate the visual target position information of the target through pixel fitting and region segmentation algorithms. The fusion decision module 40 is used to receive radar target position information obtained by radar sensors from the target object, and perform multi-sensor fusion decision-making on the radar target position information and the visual target position information through the single surround view early warning domain controller, and output the fused target object status information. The alarm control module 50 is used to determine the highest priority target regulatory warning task based on vehicle status information, the panoramic surround view image and the target object status information, through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller, and output the currently triggered alarm signal and view switching command to the vehicle display device.

[0147] The vehicle surround view warning device provided in this application, employing the vehicle surround view warning method in the above embodiments, can solve the technical problem that existing single-controller solutions lack chassis height adaptation and multi-sensor fusion, and cannot integrate to meet the target blind spot detection accuracy requirements. Compared with the prior art, the beneficial effects of the vehicle surround view warning device provided in this application are the same as those of the vehicle surround view warning method provided in the above embodiments, and other technical features in the vehicle surround view warning device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0148] This application provides a vehicle surround view warning device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the vehicle surround view warning method in the first embodiment described above.

[0149] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing a vehicle surround-view warning device according to embodiments of this application. The vehicle surround-view warning device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The vehicle surround view warning device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0150] like Figure 7As shown, the vehicle surround view warning device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the vehicle surround view warning device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the vehicle surround view warning device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show vehicle surround view warning devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0151] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0152] The vehicle surround view warning device provided in this application, employing the vehicle surround view warning method in the above embodiments, can solve the technical problem that existing single-controller solutions lack chassis height adaptation and multi-sensor fusion, and cannot integrate to meet the target blind spot detection accuracy requirements. Compared with the prior art, the beneficial effects of the vehicle surround view warning device provided in this application are the same as those of the vehicle surround view warning method provided in the above embodiments, and other technical features in this vehicle surround view warning device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0153] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0154] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0155] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A vehicle surround-view warning method, characterized in that, The vehicle surround view warning method, applied to a single surround view warning domain controller and multiple sensors, includes: Images of the vehicle's surroundings are captured by multiple wide-angle cameras, including at least one wide-angle camera covering the forward regulatory blind zone defined by the first target regulation, the rear regulatory blind zone defined by the second target regulation, and the lateral regulatory blind zone defined by the third target regulation. The single surround view warning domain controller receives the vehicle's current chassis height information, adaptively switches the current calibration parameters based on the current chassis height information, and performs distortion correction and surround view stitching on the images around the vehicle according to the current calibration parameters to generate a panoramic surround view image. The AI ​​target detection model pre-trained in the single surround-view warning domain controller is used to identify targets in the images around the vehicle, and the visual target position information of the targets is calculated through pixel fitting and region segmentation algorithms. Specifically, this includes: dividing the area where the target is located into regions based on the pixel position of the target in the image around the vehicle, and filtering the divided regions into effective regions; within the filtered effective regions, obtaining a pre-calibrated ratio coefficient between world coordinates and pixel coordinates; calculating the pixel difference between the pixel coordinates of the target and the reference pixel coordinates based on the ratio coefficient; and calculating the visual target position information of the target in the vehicle coordinate system based on the pixel difference and the ratio coefficient using a pixel fitting algorithm. The system receives radar target position information detected by radar sensors, and uses the single surround-view warning domain controller to perform multi-sensor fusion decision-making on the radar target position information and the visual target position information, outputting the fused target state information. Specifically, this includes: receiving radar coordinate information collected by the radar sensors, combining the radar sensor's installation location, unifying the radar coordinate information to the vehicle coordinate system to obtain radar target position information; associating and matching identical targets based on the visual target position information and the radar target position information, and determining the detection area type of the associated target, where the detection area type includes single-sensor detection area and multi-sensor detection overlap area; if the target is in a multi-sensor detection overlap area, then... The presence probability of a target object is fused by combining the detection rate, false alarm rate, target confidence, and presence period of each sensor within the overlapping area of ​​the multi-sensor detection, resulting in a fused presence probability. Weighting coefficients are determined based on the variance of the detection accuracy of each sensor within the overlapping area, and the target position and velocity information provided by each sensor are weighted and averaged based on these weighting coefficients to obtain fused position and velocity information. The actual velocity of the target object is calculated by compensating for the fused position and velocity information in conjunction with the vehicle's speed and wheel angle. A sliding window method is used to determine the target object as moving when the actual velocity continuously exceeds a motion judgment threshold. The fused presence probability, fused position and velocity information, and the motion state are combined to form the fused target object state information. Based on vehicle status information, the panoramic surround view image, and the target object status information, conflict arbitration is conducted through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller to determine the target regulatory warning task with the highest priority at present, and output the currently triggered alarm signal and view switching command to the vehicle display device.

2. The vehicle surround view warning method as described in claim 1, characterized in that, The acquisition of images around the vehicle via multiple wide-angle cameras includes: The vehicle collects forward and side-front image data using a first wide-angle camera and a second wide-angle camera installed on both sides of the front of the vehicle. Rear and side rear image data are collected by the third and fourth wide-angle cameras installed on both sides of the rear of the vehicle. High-resolution lateral image data is acquired by fisheye and narrow-angle cameras installed on the left and right sides of the vehicle, and lateral ranging data is acquired by millimeter-wave radar.

3. The vehicle surround view warning method as described in claim 2, characterized in that, The steps of adaptively switching the current calibration parameters based on the current chassis height information, and performing distortion correction and surround view stitching on the images around the vehicle according to the current calibration parameters to generate a panoramic surround view image include: Based on the current chassis height information, the current homography matrix, current distortion correction coefficients, and current fusion weight table are obtained by matching the pre-stored calibration parameter groups corresponding to different chassis heights. Based on the current distortion correction coefficients, the image around the vehicle is subjected to distortion correction processing, and a distortion-corrected image is output. Based on the current homography matrix, perspective transformation is performed on the distortion-reduced image to generate a bird's-eye view image; Based on the current fusion weight table, pixel-weighted fusion is performed on the overlapping areas in the bird's-eye view images corresponding to adjacent wide-angle cameras to output a panoramic view image.

4. The vehicle surround view warning method as described in claim 3, characterized in that, The step of using the pre-trained AI target detection model in the single surround-view warning domain controller to identify targets in the image around the vehicle includes: The image around the vehicle is input into a pre-trained AI object detection model, which outputs an initial set of detection boxes and the location confidence and classification confidence of each initial detection box. The initial detection box with the highest location confidence is selected as the reference box. The intersection-union ratio (IUU) of the remaining initial detection boxes and the reference box is calculated. The weighting factor of the remaining initial detection boxes is determined based on the IUU. The initial detection boxes are then weighted and fused to obtain a unique detection box. The location confidence of the unique detection box is multiplied by the classification confidence to obtain the comprehensive confidence, and the unique detection box with the comprehensive confidence higher than the preset confidence threshold is determined as a valid detection box; The valid detection boxes are matched with the target list of the current tracking algorithm. For the successfully matched valid detection boxes, the state is updated using the Kalman filter algorithm to output smooth detection boxes. The unmatched valid detection boxes are added as new targets to the target list of the tracking algorithm. Targets that have not been matched for a consecutive preset period are removed from the list to output the finally identified target.

5. The vehicle surround view warning method as described in claim 4, characterized in that, After the step of receiving radar target position information detected by radar sensors, performing multi-sensor fusion decision-making on the radar target position information and the visual target position information through the single surround-view warning domain controller, and outputting the fused target state information, the method further includes: Real-time monitoring of the health status indicators of each sensor; When any main sensor fails, the type of the failed sensor and the regulatory blind area it covers are determined based on the health status identifier, and the failed sensor information is output. Based on the information of the failed sensors, the weights of the failed sensors are reset to zero, and the weights of the failed sensors are redistributed to the remaining available sensors covering the same regulatory blind zone area according to a preset confidence level allocation rule, and the adjusted fusion weight set is output. Based on the adjusted fusion weight set, the detection data of the remaining available sensors are fused using the existence probability method, and the downgraded fused target state information is output.

6. The vehicle surround view warning method as described in claim 5, characterized in that, The steps of determining the highest priority target regulatory warning task based on vehicle status information, the panoramic surround view image, and the target object status information, through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller, and outputting the currently triggered alarm signal and view switching command to the vehicle display device include: Based on the fused position and speed information and motion state in the target object's state information, and combined with the gear, vehicle speed and turn signal status in the vehicle's state information, the warning trigger conditions of the first target regulation, the second target regulation and the third target regulation are evaluated in parallel, and corresponding regulation warning tasks are generated. When the number of regulatory warning tasks triggered at the same time is greater than or equal to two, a cross-regulatory priority arbitration mechanism is activated to arbitrate the conflict of the simultaneously triggered regulatory warning tasks according to a preset regulatory priority sequence. The regulatory priority sequence is set such that the priority of the backward warning task corresponding to the second target regulation is greater than that of the lateral warning task corresponding to the third target regulation, and the priority of the lateral warning task corresponding to the third target regulation is greater than that of the forward warning task corresponding to the first target regulation. Based on the conflict arbitration results, the highest priority target regulatory warning task is determined, and a graded alarm signal is output according to the danger level threshold corresponding to the target regulatory warning task. A perspective switching command is generated that matches the highest priority target regulatory warning task. The perspective switching command has a higher priority than the perspective switching command triggered by the normal gear position and turn signal status, so as to force the vehicle display device to switch to the forward view, rear view, side view or panoramic view image with the target highlighted corresponding to the graded alarm signal.

7. A vehicle surround view warning device, characterized in that, The vehicle surround view warning device includes: The image acquisition module is used to acquire images of the vehicle's surroundings through multiple wide-angle cameras, wherein the multiple wide-angle cameras include at least wide-angle cameras covering the forward regulatory blind spot, the rear regulatory blind spot, and the lateral regulatory blind spot; The image calibration module is used to receive the current chassis height information of the vehicle through a single surround view warning domain controller, adaptively switch the current calibration parameters based on the current chassis height information, and perform distortion correction and surround view stitching on the images around the vehicle according to the current calibration parameters to generate a panoramic surround view image. The target detection module is used to identify targets in the vehicle's surrounding images using a pre-trained AI target detection model in the single surround-view warning domain controller, and to calculate the visual target position information of the targets through pixel fitting and region segmentation algorithms. Specifically, it includes: dividing the area where the target is located into regions based on the pixel position of the target in the vehicle's surrounding images, and filtering the divided regions into effective regions; within the filtered effective regions, obtaining a pre-calibrated ratio coefficient between world coordinates and pixel coordinates; calculating the pixel difference between the target's pixel coordinates and the reference pixel coordinates based on the ratio coefficient; and calculating the visual target position information of the target in the vehicle coordinate system based on the pixel difference and the ratio coefficient using a pixel fitting algorithm. The fusion decision module receives radar target position information detected by radar sensors, and performs multi-sensor fusion decision-making by combining the radar target position information with the visual target position information through the single surround-view warning domain controller, outputting the fused target state information. Specifically, this includes: receiving radar coordinate information collected by the radar sensors, combining the radar sensor's installation position, unifying the radar coordinate information to the vehicle coordinate system to obtain radar target position information; associating and matching identical targets based on the visual target position information and the radar target position information, and determining the detection area type of the associated target, which includes single-sensor detection areas and multi-sensor detection overlap areas; if the target is in a multi-sensor detection overlap area... In the domain, the detection rate, false alarm rate, target confidence, and existence period of each sensor within the overlapping area of ​​multi-sensor detection are combined to perform existence probability fusion on the target object, obtaining a fused existence probability. Weighting coefficients are determined based on the detection accuracy variance of each sensor within the overlapping area of ​​multi-sensor detection, and the target position and velocity information provided by each sensor are weighted and averaged based on these weighting coefficients to obtain fused position and velocity information. The actual velocity of the target object is calculated by compensating the fused position and velocity information in conjunction with the vehicle's speed and wheel angle, and a sliding window method is used to determine the target object as in motion when the actual velocity continuously exceeds a motion judgment threshold. The fused existence probability, fused position and velocity information, and the motion state are combined to form the fused target object state information. The alarm control module is used to determine the highest priority target regulatory warning task based on vehicle status information, the panoramic surround view image and the target object status information, through the cross-regulatory priority arbitration mechanism in the blind spot unified warning rule integrated by the single surround view warning domain controller, and output the currently triggered alarm signal and view switching command to the vehicle display device.

8. A vehicle surround view warning device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the vehicle surround-view warning method as described in any one of claims 1 to 6.

Images