Visual enhancement method and device for vehicle and storage medium

Abstract

The embodiment of the invention provides a visual enhancement method and device for a vehicle and a storage medium. The method comprises the following steps: acquiring multi-mode environment sensing data around a vehicle through at least one distance measuring sensor and at least one optical imaging sensor carried on the vehicle; data of a global positioning system receiver, an inertial measurement unit and a wheel speed encoder are fused, and state information of the vehicle is determined through filtering; based on sensing data of the distance measuring sensor, detecting and tracking surrounding obstacles by using an obstacle detection and tracking algorithm to obtain dynamic information of the surrounding obstacles; performing real-time path planning by utilizing a path planning algorithm based on the state information, the dynamic information and the map data so as to generate a dynamic navigation path; processing sensing data of the optical imaging sensor to generate enhanced visual information; and superposing the enhanced visual information and the dynamic navigation path to form an enhanced visual interface.

Description

Technical Field

[0001] This application relates to the field of fire and rescue, and specifically to a method, device, and storage medium for visual enhancement of vehicles. Background Technology

[0002] In the field of fire and rescue, fire trucks often need to quickly enter complex disaster scenes with low visibility, such as areas with dense smoke, darkness, or numerous obstacles. Traditional vehicle vision assistance systems are mostly based on imaging and navigation technologies in standard environments. In high-dynamic, low-light scenarios such as fire scenes, they often suffer from limitations such as loss of image details and delayed obstacle avoidance. Therefore, existing vehicle vision assistance methods are difficult to apply to fire trucks. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, and storage medium for visual enhancement of vehicles.

[0004] To achieve the above objectives, a first aspect of this application provides a visual enhancement method for vehicles, comprising: Multimodal environmental perception data around the vehicle is acquired by using at least one ranging sensor and at least one optical imaging sensor mounted on the vehicle. By integrating data from the GPS receiver, inertial measurement unit, and wheel speed encoder, the vehicle's own status information is determined through filtering; Based on the sensing data from the ranging sensor, obstacle detection and tracking algorithms are used to detect and track surrounding obstacles, thereby obtaining dynamic information about the surrounding obstacles. Based on its own state information, dynamic information and map data, a path planning algorithm is used to perform real-time path planning to generate a dynamic navigation path. The path planning algorithm dynamically adjusts the cost function of the heuristic function according to the density of obstacles in the environment. The sensor data from the optical imaging sensor is subjected to coordinate transformation, target detection, and thermal imaging enhancement to generate enhanced visual information. Enhanced visual information is overlaid with dynamic navigation paths to create an enhanced visual interface.

[0005] In this embodiment, the path planning algorithm is a dynamic weighted A* algorithm. Based on its own state information, dynamic information, and map data, the path planning algorithm performs real-time path planning to generate a dynamic navigation path, including: obtaining the vehicle's current position node based on its own state information and transferring the current position node from an open list to a closed list; determining all neighboring nodes of the current position node and adding the neighboring nodes to the open list; obtaining the obstacle density of neighboring nodes based on dynamic information and determining the obstacle cost coefficient based on the obstacle density; obtaining the starting position node and the ending position node of the dynamic navigation path; determining the cost function based on the starting position node, neighboring nodes, and ending position node using the obstacle cost coefficient as a dynamic weight factor; determining the neighboring node with the smallest cost function as the next node of the dynamic navigation path and controlling the vehicle to move to the next node; repeating the above steps until the vehicle reaches the ending position node.

[0006] In this embodiment, the obstacle cost coefficient is determined according to the following formula:

[0007] in, for Obstacle cost coefficient at time step The obstacle density is Obstacle cost coefficient at time, The preset sensitivity coefficient, for Obstacle density at any given time This is the preset baseline obstacle density.

[0008] In one embodiment, the cost function is determined according to the following formula:

[0009]

[0010] in, Let cost function be For the actual path cost, As a dynamic weighting factor, For the cost of Euclidean distance, The coordinates of adjacent nodes. This refers to the endpoint location node.

[0011] In this embodiment of the application, the cost function determined based on the starting point node, adjacent nodes, and ending point node, with the obstacle cost coefficient as the dynamic weight factor, further includes: for each adjacent node, determining the influence range of each obstacle near the adjacent node based on dynamic information; when the adjacent node is located within any influence range, determining the risk weight coefficient based on the distance between the adjacent node and the obstacle corresponding to the influence area, and updating the cost function.

[0012] In the application embodiment, the cost function is updated according to the following formula:

[0013] in, For the updated cost function, For the actual path cost, For the cost of Euclidean distance, for Obstacle cost coefficient at time step This is the preset adjustment ratio coefficient. This is the risk weighting coefficient.

[0014] In the application embodiment, the vehicle's own state information is the vehicle's state vector, and the filtering is Kalman filtering. Data from the GPS receiver, inertial measurement unit, and wheel speed encoder are fused. Determining the vehicle's own state information through filtering includes: acquiring the vehicle's state vector and covariance matrix; predicting the vehicle's state information for the next moment based on the state vector, covariance matrix, and Kalman filter model to obtain a predicted state vector and predicted covariance matrix; after a preset time, acquiring the vehicle's observation vector through the GPS receiver, inertial measurement unit, and wheel speed encoder; determining the state vector and covariance matrix after the preset time based on the observation vector, predicted state vector, and predicted covariance matrix; repeating the above steps to continuously update the vehicle's own state information.

[0015] In the application embodiment, the method of determining the vehicle's own state information by fusing data from the GPS receiver, inertial measurement unit, and wheel speed encoder through Kalman filtering further includes: at the initial moment when the vehicle performs a task, the covariance matrix is ​​a preset initial covariance matrix; the observation vector of the vehicle is obtained through the GPS receiver, inertial measurement unit, and wheel speed encoder, and the observation vector is used as the state vector.

[0016] In this embodiment, the obstacle detection and tracking algorithm is the Joint Probabilistic Data Association (JPDA) algorithm. The dynamic information includes the obstacle's state vector, state covariance, and maximum velocity. Based on the perception data from the ranging sensor, the obstacle detection and tracking algorithm is used to detect and track surrounding obstacles, obtaining the dynamic information of the surrounding obstacles. This includes: calculating the association probability between the measurement values ​​and the obstacle based on the measurements from multiple ranging sensors according to the JPDA algorithm; updating the obstacle's state vector based on the association probability; updating the obstacle's state covariance matrix based on the association probability, the observation matrix, and the observation noise covariance matrix; determining the likelihood ratio based on the association probability and the likelihood function, and updating the obstacle's existence probability based on the likelihood ratio; and for each obstacle, outputting the obstacle's dynamic information when the existence probability is greater than a first preset threshold.

[0017] In this embodiment of the application, the coordinate transformation of the sensing data of the optical imaging sensor includes: determining the rotation matrix and translation vector between the coordinate systems of multiple optical imaging sensors based on the sensing data acquired by each optical imaging sensor; performing coordinate transformation on the sensing data acquired by multiple optical imaging sensors based on the rotation matrix and translation vector to unify the coordinate system of each sensing data; and fusing the sensing data after unifying the coordinate system by using a weighted average method to obtain fused sensing data.

[0018] In this embodiment of the application, the target detection includes: using the YOLOv8 target detection algorithm to extract features and detect targets in the fused sensing data to obtain a first candidate box for obstacles in the fused sensing data; and using the Faster R-CNN target detection algorithm to detect targets in the first candidate box to obtain and output a second candidate box. In this embodiment of the application, the optical imaging sensor includes at least a visible light camera and an infrared camera. The thermal imaging enhancement processing includes: performing noise reduction processing on the infrared image acquired by the infrared camera using Gaussian filtering; performing histogram equalization processing on the noise-reduced infrared image; and processing the histogram equalized infrared image based on the visible light image acquired by the visible light camera using guided filtering to obtain a thermal imaging enhancement image.

[0019] In this embodiment of the application, based on the visible light image acquired by the visible light camera, the infrared image after histogram equalization is processed by guided filtering to obtain a thermal imaging enhancement image, including: constructing a local window based on the visible light image; determining the linear coefficients of the corresponding positions of the visible light image and the infrared image within the local window; determining the pixel values ​​of the thermal imaging enhancement image within the local window based on the linear coefficients; repeating the above steps until the entire visible light image is traversed.

[0020] In this embodiment of the application, the vehicle is a fire truck.

[0021] A second aspect of this application provides a vision enhancement device for a vehicle, comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement the aforementioned vision enhancement method for vehicles.

[0022] A third aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned vision enhancement method for a vehicle.

[0023] The above technical solution achieves several improvements. First, based on data acquired from a Global Positioning System (GPS) receiver, Inertial Measurement Unit (IMU), and wheel speed encoder, Kalman filtering is used to locate and estimate the vehicle's state, providing its own state information. Second, by using perception data from multiple ranging sensors, obstacle detection and tracking algorithms are employed to fuse the perception data from multiple sensors and monitor and track surrounding obstacles, obtaining dynamic information about these obstacles and improving the accuracy and robustness of obstacle perception. Third, based on the vehicle's own state information, dynamic information, and pre-set map information, a path planning algorithm generates a dynamic navigation path, enabling the vehicle to avoid obstacles in a timely manner. Fourth, multiple optical imaging sensors acquire environmental perception data and perform coordinate transformation, target detection, and thermal imaging enhancement to generate enhanced visual information, significantly improving details in dark areas of the image. Finally, the enhanced visual information is superimposed on the dynamic navigation path to form an enhanced visual interface. With comprehensive visual enhancement support and accurate navigation and obstacle avoidance functions, the risk of collisions is effectively reduced in low-visibility environments and complex road conditions, ensuring the safety of the driver and the vehicle. Accurate path planning and rapid target detection enable the vehicle to reach the fire scene more quickly and accurately.

[0024] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 A schematic flowchart of a vision enhancement method for a vehicle according to an embodiment of this application is shown. Figure 2The schematic diagram illustrates a flow chart of yet another vision enhancement method for a vehicle according to an embodiment of this application; Figure 3 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0027] Figure 1 A schematic flowchart illustrating a vision enhancement method for a vehicle according to an embodiment of this application is shown. Figure 1 As shown in one embodiment of this application, a vision enhancement method for vehicles is provided, comprising the following steps: Step 101: Acquire multimodal environmental perception data around the vehicle by using at least one ranging sensor and at least one optical imaging sensor mounted on the vehicle.

[0028] Step 102: Integrate the data from the GPS receiver, inertial measurement unit, and wheel speed encoder, and determine the vehicle's own state information through filtering.

[0029] Step 103: Based on the sensing data from the ranging sensor, use obstacle detection and tracking algorithms to detect and track surrounding obstacles, and obtain dynamic information about the surrounding obstacles.

[0030] Step 104: Based on its own state information, dynamic information and map data, a path planning algorithm is used to perform real-time path planning to generate a dynamic navigation path. The path planning algorithm dynamically adjusts the cost function of the heuristic function according to the density of obstacles in the environment.

[0031] Step 105: Perform coordinate transformation, target detection, and thermal imaging enhancement processing on the sensing data from the optical imaging sensor to generate enhanced visual information.

[0032] Step 106: Overlay the enhanced visual information with the dynamic navigation path to form an enhanced visual interface.

[0033] Enhanced vision refers to improving the driver's visual perception ability in environments with poor re-visibility. The processor can acquire multimodal environmental perception data around the vehicle through ranging sensors and optical imaging sensors. This multimodal environmental perception data refers to a collection of environmental information provided by various sensors. The ranging sensors include at least one of LiDAR and millimeter-wave radar, and the optical imaging sensors include at least one of visible light cameras and infrared cameras. Then, based on the perception data from the ranging sensors, the processor can use obstacle detection and tracking algorithms to detect and track surrounding obstacles, obtaining dynamic information about these obstacles. This dynamic information includes the obstacle's state vector, state covariance, and maximum velocity; the state covariance characterizes the uncertainty in obstacle prediction. On the other hand, the processor can fuse data from a GPS receiver, an IMU, and wheel speed encoders, and determine the vehicle's own state information through filtering. The GPS receiver provides the vehicle's absolute position and velocity information; the IMU measures three-axis acceleration and angular velocity for short-term attitude estimation and motion prediction; the wheel speed encoders provide the vehicle's wheel speed; and the vehicle's own state information includes its position and velocity. Furthermore, the processor can perform real-time path planning based on its own state information, dynamic information, and map data to generate a dynamic navigation path. The path planning algorithm dynamically adjusts the cost function of the heuristic function according to the density of obstacles in the environment. The processor can also perform coordinate transformation, target detection, and thermal imaging enhancement processing on the sensor data from the optical imaging sensor to generate enhanced visual information. This enhanced visual information is then overlaid with the dynamic navigation path to form an enhanced visual interface.

[0034] In the application embodiment, the self-state information is the vehicle's state vector, and the filtering is Kalman filtering. The data from the GPS receiver, IMU, and wheel speed encoder are fused. The vehicle's self-state information is determined by filtering as follows: at the initial moment when the vehicle performs the task, the covariance matrix is ​​a preset initial covariance matrix; the observation vector of the vehicle is obtained through the GPS receiver, IMU, and wheel speed encoder, and the observation vector is used as the state vector.

[0035] In one embodiment, the vehicle's self-state information is a state vector. Fusing data from a GPS receiver, an IMU, and a wheel speed encoder, determining the vehicle's self-state information through Kalman filtering further includes: acquiring the vehicle's state vector and covariance matrix; and predicting the vehicle's state information for the next moment based on the state vector, covariance matrix, and Kalman filter model to obtain a predicted state vector and a predicted covariance matrix. Specifically, the predicted state vector and predicted covariance matrix are determined based on the following formulas:

[0036]

[0037] in, for Always The predicted state vector at time , Here is the state transition matrix. for The real-time state vector at time t, To control the input matrix, For control inputs, including vehicle steering control signals and acceleration control signals, for Always The prediction covariance matrix at time 1, for The covariance matrix at time t, This is the process noise covariance matrix. After a preset time, the vehicle's observation vector is acquired through a GPS receiver, an inertial measurement unit (IMU), and a wheel speed encoder. Based on the observation vector, the predicted state vector, and the predicted covariance matrix, the state vector and covariance matrix after the preset time are determined. Specifically, the state vector and covariance matrix after the preset time are determined according to the following formulas:

[0038]

[0039]

[0040] in, For Kalman gain, for Always The predicted covariance matrix at time t, For the observation matrix, To observe the noise covariance matrix, for The real-time state vector at time t, for Always The predicted state vector at time , for The observation vector at time [time], Let be the covariance matrix at time t. Repeat the above steps to continuously update the vehicle's own state information.

[0041] In one embodiment, the obstacle detection and tracking algorithm is a Joint Probabilistic Data Association (JPDA) algorithm. Dynamic information includes the obstacle's state vector, state covariance, and maximum velocity. Based on the sensing data from the ranging sensor, the JPDA algorithm is used to detect and track surrounding obstacles, obtaining dynamic information about these obstacles. This includes calculating the association probability between measurements from multiple ranging sensors and the obstacle, according to the JPDA algorithm.

[0042] in, For the first The measured value and the first The association probability of each of the obstacles For the first The measured value and the first The likelihood function of the obstacle. For the first The measured value and the first The likelihood function of each obstacle reflects the degree of matching between the measured value and the target. The total number of the measured values. Clutter density describes the number of false targets in the environment. The obstacle's state vector is updated based on the association probability.

[0043] in for Time of the first The state vector of each obstacle. for Time of the first The first measurement value and the first The associated probability of each obstacle for The number of measurements at any given time. The pseudo-inverse of the observation matrix. for The first moment There are several measurement values. The relationship between the observation matrix, the measurement values, and the true state vector of the obstacle is as follows:

[0044] in, For measurement value, For the observation matrix, For the true state vector, Let be the covariance matrix of Gaussian noise. Update the obstacle's state covariance matrix based on the association probability, observation matrix, and observation noise covariance matrix; determine the likelihood ratio based on the association probability and likelihood function, and update the obstacle's existence probability based on the likelihood ratio:

[0045] in, for Time of the first The probability of the existence of an obstacle for Time of the first The probability of the existence of an obstacle For time k, the first The likelihood ratio of each obstacle. For each obstacle, if its probability of existence is greater than a first preset threshold, output the dynamic bounding box of the obstacle, which includes the dynamic information of the obstacle:

[0046] in Let i be the dynamic bounding box of the i-th obstacle at time k. Let i be the state vector of the i-th obstacle at time k. Let be the state covariance of the i-th obstacle at time k. Let be the maximum speed of the i-th obstacle at time k.

[0047] In one embodiment, the path planning algorithm is a dynamic weighted A-Star algorithm. Based on its own state information, dynamic information, and map data, the path planning algorithm performs real-time path planning to generate a dynamic navigation path, including: obtaining the vehicle's current position node based on its own state information and transferring the current position node from an open list to a closed list; determining all neighboring nodes of the current position node and adding the neighboring nodes to the open list; obtaining the obstacle density of neighboring nodes based on dynamic information and determining the obstacle cost coefficient based on the obstacle density; obtaining the starting position node and the ending position node of the dynamic navigation path; determining the cost function based on the starting position node, neighboring nodes, and ending position node using the obstacle cost coefficient as a dynamic weight factor; determining the neighboring node with the smallest cost function as the next node of the dynamic navigation path and controlling the vehicle to move to the next node; repeating the above steps until the vehicle reaches the ending position node.

[0048] Specifically, in one embodiment, the obstacle cost coefficient is determined according to the following formula:

[0049] in, for Obstacle cost coefficient at time step The obstacle density is Obstacle cost coefficient at time, The preset sensitivity coefficient, for Obstacle density at any given time The preset baseline obstacle density. In one embodiment, the cost function is determined according to the following formula:

[0050]

[0051] in, Let cost function be For the actual path cost, As a dynamic weighting factor, For the cost of Euclidean distance, The coordinates of adjacent nodes. This is the endpoint node.

[0052] In one embodiment, determining the cost function based on the starting node, neighboring nodes, and ending node, using obstacle cost coefficients as dynamic weighting factors, further includes: for each neighboring node, determining the influence range of each obstacle near the neighboring node based on dynamic information; and, if a neighboring node is within any influence range, determining a risk weight coefficient based on the distance between the neighboring node and the obstacle corresponding to the influence area, and updating the cost function. Specifically, in one embodiment, the cost function is updated according to the following formula:

[0053] in, For the updated cost function, For the actual path cost, For the cost of Euclidean distance, for Obstacle cost coefficient at time step This is the preset adjustment ratio coefficient. This is the risk weighting coefficient.

[0054] In one embodiment, coordinate transformation of the sensing data from the optical imaging sensor includes: determining a rotation matrix and translation vector between the coordinate systems of multiple optical imaging sensors based on the sensing data acquired by each optical imaging sensor; performing coordinate transformation on the sensing data acquired by the multiple optical imaging sensors based on the rotation matrix and translation vector to unify the coordinate system of each sensing data; and fusing the unified coordinate system sensing data using a weighted average method to obtain fused sensing data. For example, in one embodiment, the optical imaging sensor includes a lidar, a visible light camera, and an infrared camera. The lidar can acquire high-precision three-dimensional spatial information to accurately identify the position and shape of the target; the visible light camera provides rich color and texture information, which helps to identify the appearance features of the target; and the infrared camera can detect the thermal radiation information of the target, having unique advantages in low-visibility environments. Based on the sensing data acquired by each optical imaging sensor, the rotation matrix and translation vector between the coordinate systems of the optical imaging sensors are obtained using a calibration board or a 3D point cloud matching algorithm, and then coordinate transformation is performed.

[0055]

[0056] in, Let P be the coordinates of point P in the visible light camera coordinate system. This is the rotation matrix between the visible light camera coordinate system and the infrared camera coordinate system. Let P be the coordinates of point P in the infrared camera coordinate system. This is the translation vector between the visible light camera coordinate system and the infrared camera coordinate system. This is the rotation matrix between the visible light camera coordinate system and the lidar coordinate system. This represents the translation vector between the visible light camera coordinate system and the lidar coordinate system. A weighted averaging method is used to fuse data from different sensors, improving data accuracy and robustness.

[0057] in, , , These are measurements of the same target taken by a visible light camera, an infrared camera, and a lidar, respectively. These are the measured values ​​after fusion. , , As weight, Furthermore, the weights are determined based on the measurement accuracy and reliability of each sensor. For example, if the lidar has high accuracy in distance measurement, its corresponding weight... The weight can be set to a larger value; while visible light cameras have advantages in color and texture recognition, and their weight... It can be adjusted appropriately according to the specific application scenario.

[0058] In one embodiment, object detection includes: using the YOLOv8 object detection algorithm to perform feature extraction and object detection on the fused sensing data through a convolutional neural network to obtain a first candidate bounding box for obstacles in the fused sensing data. During the processing, the loss function includes classification loss, localization loss, and confidence loss, with the specific formulas as follows:

[0059] in, For loss function, These are the weighting coefficients for the localization loss. It is the number of grids in the image. It is the number of bounding boxes predicted for each grid cell. This indicates whether the i-th bounding box of the i-th grid contains the target (1 if it contains the target, 0 otherwise). These are the center coordinates of the bounding box. These are the center coordinates of the real target. YOLOv8 is characterized by its fast detection speed and high accuracy, enabling real-time target detection in images. The Faster R-CNN object detection algorithm is used to detect targets in the first candidate box, and Focal Loss is introduced to address the imbalance of training data for different detected targets, thus obtaining and outputting the second candidate box. The Focal Loss is:

[0060] in, For Focal Loss, For classification probability, To adjust the weights of easy and difficult samples, To balance the weights of positive and negative samples, Faster R-CNN, which has the advantage of high detection accuracy, combines YOLOv8 and Faster R-CNN to improve the accuracy of target detection while maintaining detection speed.

[0061] In one embodiment, the optical imaging sensor includes at least a visible light camera and an infrared camera. The thermal imaging enhancement processing includes: performing noise reduction processing on the infrared image acquired by the infrared camera using Gaussian filtering; performing histogram equalization processing on the noise-reduced infrared image; and processing the histogram equalized infrared image based on the visible light image acquired by the visible light camera using guided filtering to obtain a thermal imaging enhanced image. Specifically, in one embodiment, performing noise reduction processing on the infrared image acquired by the infrared camera using Gaussian filtering includes: generating a filter kernel using a Gaussian function; and performing a weighted average of each pixel and its neighboring pixels in the infrared image based on the filter kernel, wherein the weight of each pixel is calculated based on the following formula:

[0062] in, These are the coordinates of pixels in the image. The weights of the pixel (x, y) are... and These are the mean values ​​in the x and y directions, respectively. It is the standard deviation of the Gaussian distribution, used to control the decay rate of the Gaussian function.

[0063] In one embodiment, processing the histogram-equalized infrared image using guided filtering based on the visible light image acquired by a visible light camera to obtain a thermally enhanced image includes: constructing a local window based on the visible light image; and determining the linear coefficients at corresponding positions in the visible light and infrared images within the local window.

[0064] in, , The linear coefficients are... In pixels The local window centered on the center, For the visible light image at pixel points pixel values, For the infrared image at pixel points pixel values, This is a regularization parameter. The pixel values ​​of the thermal imaging enhancement image within a local window are determined based on linear coefficients.

[0065] in, and In pixels The linear coefficients of the local window centered on the given point. The infrared image at the pixel point i pixel values, For the thermal imaging enhancement image at pixel points i The pixel values; repeat the above steps until the entire visible light image has been traversed.

[0066] In one embodiment, the vehicle is a fire truck.

[0067] Through the above embodiments, by integrating multiple sensors and employing advanced multi-sensor fusion technologies, such as Kalman filtering and the JPDA algorithm, deep fusion of data from different sensors is achieved. This fully leverages the advantages of various sensors, including LiDAR, millimeter-wave radar, infrared cameras, and visible light cameras, improving the accuracy and robustness of environmental perception and effectively addressing various challenges in complex environments. For low-visibility environments, this solution combines the thermal radiation detection capabilities of infrared cameras with the rich information from visible light cameras, and employs thermal imaging enhancement algorithms, such as histogram equalization and guided filtering, significantly improving target recognition capabilities and dark area details under interference factors such as smoke and dust. This provides fire truck drivers with clear and reliable visual information, greatly enhancing driving safety. The dynamic weight A* algorithm can adjust the cost function of path planning in real time according to environmental changes (such as obstacle density), achieving efficient path planning in dynamic and complex environments. Meanwhile, by combining dynamic obstacle information provided by the JPDA algorithm, obstacles can be avoided in a timely manner, ensuring that fire trucks arrive at the rescue scene safely and quickly. Two object detection algorithms, YOLOv8 and Faster R-CNN, are used. YOLOv8 achieves fast object detection, meeting real-time requirements; Faster R-CNN performs secondary verification of candidate boxes, improving the accuracy of small object detection. The combination of these two algorithms improves the accuracy of object detection while maintaining detection speed. Through comprehensive visual enhancement support and accurate navigation and obstacle avoidance functions, the collision risk of fire trucks in low-visibility environments and complex road conditions is effectively reduced, ensuring the safety of drivers and vehicles. Furthermore, accurate path planning and rapid object detection enable fire trucks to reach the fire scene more quickly and accurately.

[0068] Figure 1 This is a flowchart illustrating a vision enhancement method for a vehicle in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0069] In one embodiment, such as Figure 2 As shown, the vehicle is a fire truck, and the visual enhancement methods used for the vehicle include: The tracking subsystem determines the fire truck's own state information and the dynamic information of surrounding obstacles: At the initial moment of the fire truck's mission, the covariance matrix is ​​a preset initial covariance matrix; the observation vector of the fire truck is acquired through a GPS receiver, inertial measurement unit (IMU), and wheel speed encoder, and this observation vector is used as the state vector. The fire truck's state vector and covariance matrix are then acquired; based on the state vector, covariance matrix, and Kalman filter model, the fire truck's state information for the next moment is predicted to obtain the predicted state vector and predicted covariance matrix. After a preset time, the fire truck's observation vector is acquired again through the GPS receiver, IMU, and wheel speed encoder; based on the observation vector, predicted state vector, and predicted covariance matrix, the state vector and covariance matrix after the preset time are determined; the above steps are repeated to continuously update the fire truck's own state information. Based on the Joint Probability Data Association (JPDA) algorithm, the association probability between the measurement values ​​and obstacles is calculated based on the measurement values ​​of LiDAR and millimeter-wave radar. The state vector of the obstacle is updated based on the association probability. The likelihood ratio is determined based on the association probability and the likelihood function. The existence probability of the obstacle is updated based on the likelihood ratio. For each obstacle, if the existence probability is greater than a first preset threshold, the dynamic bounding box of the obstacle is output, which includes the dynamic information of the obstacle.

[0070] The navigation subsystem utilizes a dynamic weighted A-Star algorithm for real-time path planning to generate a dynamic navigation path: Based on its own state information, the fire truck's current location node is obtained and moved from an open list to a closed list; all neighboring nodes of the current location node are identified and added to the open list; based on dynamic information, the obstacle density of neighboring nodes is obtained, and an obstacle cost coefficient is determined based on the obstacle density; the starting and ending locations of the dynamic navigation path are obtained; using the obstacle cost coefficient as a dynamic weighting factor, a cost function is determined based on the starting, neighboring, and ending locations; for each neighboring node, the influence range of each obstacle near the neighboring node is determined based on dynamic information; if a neighboring node is within any influence range, a risk weight coefficient is determined based on the distance between the neighboring node and the obstacle corresponding to the influence area, and the cost function is updated; the neighboring node with the smallest cost function is determined as the next node in the dynamic navigation path, and the fire truck is controlled to move to the next node; the above steps are repeated until the fire truck reaches the ending location node.

[0071] The visual enhancement subsystem performs coordinate transformation, target detection, and thermal imaging enhancement on the perceived data from the optical imaging sensors to generate enhanced visual information. The optical imaging sensors include LiDAR, a visible light camera, and an infrared camera. LiDAR acquires high-precision 3D spatial information, accurately identifying the position and shape of targets. The visible light camera provides rich color and texture information, aiding in identifying the appearance features of targets. The infrared camera detects the thermal radiation information of targets, offering unique advantages in low-visibility environments. Based on the perceived data acquired by each optical imaging sensor, a calibration board or 3D point cloud matching algorithm is used to obtain the rotation matrix and translation vector between the coordinate systems of the optical imaging sensors, and coordinate transformation is performed. A weighted averaging method is used to fuse the data from different sensors, improving the accuracy and robustness of the data. The YOLOv8 target detection algorithm is used to extract features and detect targets from the fused perceived data using a convolutional neural network, obtaining first candidate boxes for obstacles in the fused perceived data. The Faster R-CNN target detection algorithm is then used to detect targets within the first candidate boxes, and Focal Loss is introduced to address the imbalance of training data for different detected targets, resulting in and outputting second candidate boxes. The infrared images acquired by the infrared camera are denoised using Gaussian filtering; histogram equalization is performed on the denoised infrared images; a local window is constructed based on the visible light image; the linear coefficients of the corresponding positions of the visible light image and the infrared image within the local window are determined; the pixel values ​​of the thermal imaging enhancement image within the local window are determined based on the linear coefficients; the above steps are repeated until the entire visible light image is traversed.

[0072] In one embodiment, a vision enhancement device for a vehicle is provided, wherein: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement the aforementioned vision enhancement method for vehicles.

[0073] The vision enhancement device for vehicles includes a processor and a memory. The processor contains a kernel that retrieves corresponding program units from the memory. One or more kernels can be configured, and the vision enhancement method for vehicles can be implemented by adjusting the kernel parameters.

[0074] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0075] This application provides a storage medium storing a program that, when executed by a processor, implements the aforementioned vision enhancement method for vehicles.

[0076] This application provides a processor for running a program, wherein the program executes the above-described visual enhancement method for vehicles.

[0077] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor A01, a network interface A02, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When executed by the processor A01, the computer program B02 implements a vision enhancement method for vehicles.

[0078] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0079] This application provides a computer (electronic) device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the above-mentioned vision enhancement methods for vehicles.

[0080] This application also provides a computer program product that, when executed on a data processing device, is adapted to execute a program that initializes a method for visual enhancement of a vehicle.

[0081] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0086] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0087] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0088] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0089] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A visual enhancement method for vehicles, characterized in that, The method includes: Multimodal environmental perception data around the vehicle is acquired by at least one ranging sensor and at least one optical imaging sensor mounted on the vehicle. By integrating data from a GPS receiver, an inertial measurement unit, and a wheel speed encoder, the vehicle's own state information is determined through filtering. Based on the sensing data from the ranging sensor, obstacle detection and tracking algorithms are used to detect and track surrounding obstacles, thereby obtaining dynamic information about the surrounding obstacles. Based on the self-state information, the dynamic information, and the map data, a path planning algorithm is used to perform real-time path planning to generate a dynamic navigation path, wherein the path planning algorithm dynamically adjusts the cost function of the heuristic function according to the density of obstacles in the environment. The perceived data from the optical imaging sensor is subjected to coordinate transformation, target detection, and thermal imaging enhancement processing to generate enhanced visual information. The enhanced visual information is overlaid with the dynamic navigation path to form an enhanced visual interface.

2. The visual enhancement method for vehicles according to claim 1, characterized in that, The path planning algorithm is a dynamic weighted A-Star algorithm. The step of generating a dynamic navigation path in real-time using the path planning algorithm based on the self-state information, the dynamic information, and map data includes: The current location node of the vehicle is obtained based on its own state information, and the current location node is transferred from the open list to the closed list; Determine all neighboring nodes of the current position node, and add the neighboring nodes to the open list; Based on the dynamic information, the obstacle density of the adjacent nodes is obtained, and the obstacle cost coefficient is determined based on the obstacle density. Obtain the start and end point locations of the dynamic navigation path; Using the obstacle cost coefficient as a dynamic weighting factor, the cost function is determined based on the starting point node, the adjacent nodes, and the ending point node; The neighboring node with the minimum cost function is determined as the next node of the dynamic navigation path, and the vehicle is controlled to move to the next node; Repeat the above steps until the vehicle reaches the destination node.

3. The visual enhancement method for vehicles according to claim 2, characterized in that, The obstacle cost coefficient is determined according to the following formula: in, for The obstacle cost coefficient at time t, The density of the obstacle is Obstacle cost coefficient at time, The preset sensitivity coefficient, for The obstacle density at time [time], This is the preset baseline obstacle density.

4. The visual enhancement method for vehicles according to claim 2, characterized in that, The cost function is determined according to the following formula: in, Let the cost function be... For the actual path cost, The dynamic weighting factor, For the cost of Euclidean distance, Let be the coordinates of the adjacent nodes. This is the endpoint node.

5. The visual enhancement method for vehicles according to claim 2, characterized in that, The step of determining the cost function based on the starting point node, the adjacent nodes, and the ending point node, using the obstacle cost coefficient as a dynamic weighting factor, further includes: For each of the adjacent nodes, the influence range of each obstacle near the adjacent node is determined based on the dynamic information; When the adjacent node is located within any one of the influence ranges, the risk weight coefficient is determined based on the distance between the adjacent node and the obstacle corresponding to the influence range, and the cost function is updated.

6. The visual enhancement method for vehicles according to claim 5, characterized in that, The cost function is updated according to the following formula: in, The updated cost function, For the actual path cost, For the cost of Euclidean distance, for The obstacle cost coefficient at time t, This is the preset adjustment ratio coefficient. The risk weighting coefficient is mentioned above.

7. The visual enhancement method for vehicles according to claim 1, characterized in that, The self-state information is the vehicle's state vector, the filtering is Kalman filtering, and the process of fusing data from the GPS receiver, inertial measurement unit, and wheel speed encoder to determine the vehicle's self-state information through filtering includes: Obtain the state vector and covariance matrix of the vehicle; Based on the state vector, covariance matrix, and Kalman filter model, the state information of the vehicle at the next moment is predicted to obtain the predicted state vector and the predicted covariance matrix. After a preset time, the observation vector of the vehicle is obtained through the global positioning system receiver, inertial measurement unit and wheel speed encoder; The state vector and covariance matrix after the preset time are determined based on the observation vector, the predicted state vector, and the predicted covariance matrix. Repeat the above steps to continuously update the vehicle's own status information.

8. The visual enhancement method for vehicles according to claim 7, characterized in that, The process of fusing data from the GPS receiver, inertial measurement unit, and wheel speed encoder, and then filtering it to determine the vehicle's own state information, also includes: At the initial moment when the vehicle performs the task, the covariance matrix is ​​a preset initial covariance matrix; The observation vector of the vehicle is obtained through the GPS receiver, inertial measurement unit and wheel speed encoder, and the observation vector is used as the state vector.

9. The visual enhancement method for vehicles according to claim 1, characterized in that, The obstacle detection and tracking algorithm is the Joint Probabilistic Data Association (JPDA) algorithm. The dynamic information includes the obstacle's state vector, the obstacle's state covariance, and the obstacle's maximum velocity. Based on the sensing data from the ranging sensor, the obstacle detection and tracking algorithm is used to detect and track surrounding obstacles, obtaining the dynamic information of the surrounding obstacles, including: The association probability between the measured values ​​and the obstacle is calculated based on the joint probability data association JPDA algorithm and the measured values ​​of multiple ranging sensors. The state vector of the obstacle is updated based on the association probability; The state covariance matrix of the obstacle is updated based on the association probability, the observation matrix, and the observation noise covariance matrix. The likelihood ratio is determined based on the association probability and the likelihood function, and the existence probability of the obstacle is updated based on the likelihood ratio; For each obstacle, if the probability of its existence is greater than a first preset threshold, the dynamic information of the obstacle is output.

10. The visual enhancement method for vehicles according to claim 1, characterized in that, The coordinate transformation of the sensing data from the optical imaging sensor includes: Based on the sensing data acquired by each of the optical imaging sensors, determine the rotation matrix and translation vector between the coordinate systems of the multiple optical imaging sensors; Based on the rotation matrix and the translation vector, the coordinate transformation is performed on the sensing data acquired by the multiple optical imaging sensors to unify the coordinate system of each sensing data. The sensed data after unifying the coordinate system is fused using a weighted average method to obtain fused sensed data.

11. The method for enhancing vehicle vision according to claim 1 or 10, characterized in that, The target detection includes: The YOLOv8 object detection algorithm is used to extract features and detect objects in the fused sensing data to obtain the first candidate bounding box for obstacles in the fused sensing data; The Faster R-CNN object detection algorithm is used to detect the objects in the first candidate box to obtain and output the second candidate box.

12. The visual enhancement method for vehicles according to claim 1, characterized in that, The optical imaging sensor includes at least a visible light camera and an infrared camera, and the thermal imaging enhancement processing includes: The infrared images acquired by the infrared camera are noise-reducing processed using Gaussian filtering; Histogram equalization is performed on the denoised infrared image. Based on the visible light image acquired by the visible light camera, the infrared image after histogram equalization is processed by guided filtering to obtain a thermal imaging enhancement image.

13. The visual enhancement method for vehicles according to claim 12, characterized in that, The process of processing the histogram-equalized infrared image based on the visible light image acquired by the visible light camera, through guided filtering, to obtain a thermal imaging enhanced image includes: A local window is constructed based on the visible light image; Determine the linear coefficients at corresponding positions of the visible light image and the infrared image within the local window; The pixel values ​​of the thermal imaging enhancement image within the local window are determined based on the linear coefficient. Repeat the above steps until the entire visible light image has been traversed.

14. The visual enhancement method for a vehicle according to claim 1, wherein the vehicle is a fire truck.

15. A vision enhancement device for a vehicle, characterized in that, include: The memory is configured to store instructions; The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the vision enhancement method for a vehicle according to any one of claims 1 to 14.

16. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, the instruction causes the processor to be configured to perform the vision enhancement method for a vehicle according to any one of claims 1 to 14.

Images