Unmanned fire extinguishing vehicle target screening method and system based on multi-sensor fusion

Abstract

The invention discloses an unmanned fire extinguishing vehicle target screening method and system based on multi-sensor fusion. According to the method, three different sensors, namely a laser radar, a camera and a millimeter-wave radar, are combined to increase the recognition weight of each sensor in different scenes, and obstacle fusion information is output to a regulation controller to carry out target screening prediction processing. And if the obstacle is predicted to be collided, executing a braking instruction through an unmanned vehicle IPB mechanism to perform obstacle braking processing in time. The braking instruction is issued in time for the obstacle predicted to have the collision risk, the IPB executes the braking instruction to complete the braking of the unmanned vehicle, and the driving safety and task execution success rate of the unmanned vehicle are improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and system for target screening of unmanned fire trucks based on multi-sensor fusion. Background Technology

[0002] In the field of autonomous driving, target selection technology for unmanned fire trucks is a crucial element in ensuring their safe operation and efficient task completion. Because lidar (LiDAR) possesses characteristics such as high resolution, good concealment, strong resistance to active interference, small size, and light weight, it can accurately identify obstacle information for unmanned vehicles in most operating conditions. Therefore, existing target selection schemes for unmanned vehicles mostly rely on a single lidar sensor. The lidar detects obstacles, identifies each obstacle within its detection range, and outputs the information to the traffic control system (PCS) for target selection. The PCS then uses the vehicle's attitude and trajectory to predict whether there is a collision risk with the unmanned vehicle. If a collision risk is detected, an IPB (Inputting Process Control) is sent to request braking and deceleration to ensure the unmanned vehicle's safe operation.

[0003] Existing technical solutions are only suitable for scenarios with simple obstacles and low task complexity. Due to the characteristics of LiDAR, it cannot identify the characteristics of obstacles, such as people, vehicles, and trees. When obstacles with different characteristics appear simultaneously, it cannot handle them separately. Target selection can only use the same logic and parameters to judge obstacle collision risk, which increases the safety risks of autonomous vehicles and reduces their scenario passability. In this scenario, cameras can be used to identify obstacle characteristics and improve the safety of autonomous vehicles.

[0004] In firefighting environments, smoke may be generated. Due to the characteristics of lidar, excessive smoke concentration can affect obstacle recognition, reducing the success rate of firefighting missions. Introducing millimeter-wave radar in this scenario can reduce the impact of smoke on sensor obstacle recognition, improving the mission success rate. Summary of the Invention

[0005] Based on this, this application provides a target screening method and system for unmanned fire trucks based on multi-sensor fusion. This application combines the characteristics and advantages / disadvantages of three different sensors—LiDAR, camera, and millimeter-wave radar—and increases the recognition weight of each sensor in different scenarios, accurately transmitting various obstacles encountered during the journey to the control system for target screening. The control system uses the fused obstacle target data processed by the perception side for logical processing. For obstacles predicted to pose a collision risk, a braking command is promptly issued. The IPB executes the braking command to bring the unmanned vehicle to a stop, increasing the driving safety of the unmanned vehicle and the success rate of mission execution.

[0006] Firstly, a target selection method for unmanned fire trucks based on multi-sensor fusion is provided, the method comprising:

[0007] The obstacle information identified by lidar, camera and millimeter-wave radar is aggregated and judged to determine whether there is smoke in the current environment;

[0008] Obstacle information is processed based on the smoke detection results of the current environment; when smoke is detected in the environment, the fusion weights of each sensor are adjusted; when no smoke is detected in the environment, the preset weights of each sensor remain unchanged.

[0009] Collision risk assessment is performed using processed obstacle information; this includes target selection based on the vehicle's attitude and target selection based on the autonomous vehicle's planned path.

[0010] If a collision risk is determined, the IPB mechanism will execute a braking command to bring the vehicle to a stop.

[0011] Optionally, in processing obstacle information based on the current environment's smoke judgment results, when smoke is detected in the environment, the weight of the millimeter-wave radar is adjusted to a preset high weight value, while the weights of the camera and lidar are adjusted to preset low weight values ​​respectively.

[0012] Optionally, target selection and judgment can be performed based on the vehicle's attitude, specifically including:

[0013] Based on the autonomous vehicle's attitude, the vehicle's trajectory is fitted by multiple polynomials to predict its position in the future.

[0014] Based on the predicted location of the autonomous vehicle and obstacle information, determine whether there is a risk of collision between the autonomous vehicle and the obstacle.

[0015] Optionally, target selection and judgment can be performed based on the autonomous vehicle's planned path, specifically including:

[0016] Based on the planned path of the autonomous vehicle, determine the area where the autonomous vehicle can drive;

[0017] The obstacle information is compared with the autonomous vehicle's drivable area to determine whether the obstacle is on the autonomous vehicle's driving path.

[0018] Optionally, obstacle information identified by lidar, cameras, and millimeter-wave radar is aggregated and judged, including:

[0019] The obstacle information identified by lidar, cameras and millimeter-wave radar is converted into a unified data format.

[0020] The converted obstacle information is then fused to generate comprehensive obstacle information.

[0021] Optionally, if a collision risk is determined, the IPB mechanism executes a braking command to bring the vehicle to a stop, specifically including:

[0022] After determining that there is a risk of collision, a braking command is generated and sent to the IPB mechanism;

[0023] After receiving the braking command, the IPB mechanism calculates the required deceleration to ensure that the autonomous vehicle stops within a safe distance;

[0024] The IPB mechanism controls the braking system to apply corresponding braking force based on the calculated deceleration;

[0025] Monitor the braking status of the unmanned vehicle to confirm that it has successfully stopped.

[0026] Secondly, a target screening system for unmanned fire trucks based on multi-sensor fusion is provided, the system comprising:

[0027] The summary judgment module is used to summarize and judge the obstacle information identified by LiDAR, camera and millimeter wave radar, and to determine whether there is smoke in the current environment;

[0028] The processing module is used to process obstacle information according to the smoke judgment results of the current environment; when smoke is detected in the environment, the fusion weights of each sensor are adjusted; when no smoke is detected in the environment, the preset weights of each sensor are kept unchanged.

[0029] The risk assessment module is used to assess collision risks using processed obstacle information; the collision risk assessment includes predictive target selection based on the vehicle's attitude and target selection based on the autonomous vehicle's planned path.

[0030] If the braking processing module determines that there is a risk of collision, it executes a braking command through the IPB mechanism to perform braking processing.

[0031] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the unmanned fire truck target screening method described in any of the first aspects above.

[0032] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the unmanned fire truck target screening method described in any of the first aspects above.

[0033] Fifthly, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the unmanned fire truck target screening method described in any of the first aspects above.

[0034] The beneficial effects of the technical solutions provided in this application include at least the following:

[0035] (1) In most scenarios, specific filtering logic thresholds can be applied to obstacles with different characteristics, which can increase the scenario passability of unmanned vehicles;

[0036] (2) The obstacle identification information is more accurate, which can increase the safety of unmanned vehicles during driving missions;

[0037] (3) In the case of smoke, multiple sensors can reduce the impact of high concentrations of smoke on lidar and increase the success rate of unmanned vehicle firefighting missions. Attached Figure Description

[0038] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0039] Figure 1 A flowchart illustrating the steps of the unmanned fire truck target screening method provided in this application embodiment;

[0040] Figure 2 A flowchart illustrating the overall process of the unmanned fire truck target screening method provided in this application embodiment;

[0041] Figure 3 This application provides an architecture diagram of an unmanned fire truck target screening system.

[0042] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] In the description of this invention, the terms "comprising," "having," and any variations are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations of the inventive concept.

[0045] To facilitate understanding of this embodiment, a method for target screening of unmanned fire trucks based on multi-sensor fusion, disclosed in this application embodiment, will first be described in detail.

[0046] This invention proposes a target screening scheme for unmanned fire trucks based on multi-sensor fusion, applicable to various mission scenarios. This scheme combines the characteristics, advantages, and disadvantages of three different sensors: lidar, cameras, and millimeter-wave radar. It increases the recognition weight of each sensor in different scenarios, accurately transmitting various obstacles encountered during the journey to the control system for target screening. The control system uses the fused obstacle target data processed by the perception side for logical processing. For obstacles predicted to pose a collision risk, a braking command is promptly issued. The Intermediate Vehicle Braking (IPB) executes the braking command to bring the unmanned vehicle to a stop, increasing the vehicle's driving safety and mission success rate.

[0047] Please refer to Figure 1 The document illustrates a flowchart of a target screening method for unmanned fire trucks based on multi-sensor fusion, as provided in an embodiment of this application. The method may include the following steps:

[0048] 1. Summarize and judge the obstacle information identified by lidar, camera and millimeter-wave radar to determine whether there is smoke in the current environment.

[0049] 2. Based on the current smoke situation, process the obstacle information accordingly.

[0050] Specifically, when smoke is detected in the environment, the fusion weights of each sensor are adjusted; when no smoke is detected, the preset weights of each sensor remain unchanged. In a smoke-filled environment, the weight of the millimeter-wave radar is adjusted to a preset high weight value, while the weights of the camera and lidar are adjusted to preset low weight values ​​respectively; in a smoke-free environment, the weights of each sensor remain at their preset default weight values.

[0051] 3. Use the processed obstacle information to assess collision risk.

[0052] The collision risk assessment includes predicting target selection based on the vehicle's attitude and selecting target based on the autonomous vehicle's planned path.

[0053] The prediction of targets is based on the vehicle's attitude. Specifically, this includes using the vehicle's attitude as a baseline and fitting the vehicle's trajectory with multiple polynomials to predict the vehicle's position in the future; and judging whether there is a risk of collision between the vehicle and obstacles based on the predicted vehicle position and obstacle information.

[0054] The target selection and judgment based on the autonomous vehicle's planned path specifically includes: determining the autonomous vehicle's drivable area based on the autonomous vehicle's planned path; comparing obstacle information with the autonomous vehicle's drivable area to determine whether the obstacle is on the autonomous vehicle's driving path.

[0055] 4. If a collision risk is determined, the IPB mechanism will execute a braking command to bring the vehicle to a stop.

[0056] Specifically, after determining that there is a collision risk, a braking command is generated and sent to the IPB mechanism; after receiving the braking command, the IPB mechanism calculates the required deceleration to ensure that the autonomous vehicle stops within a safe distance;

[0057] Based on the calculated deceleration, the IPB mechanism controls the braking system to apply corresponding braking force; it monitors the braking status of the unmanned vehicle and confirms that the unmanned vehicle has successfully stopped.

[0058] In summary, this invention provides a target screening method for unmanned fire trucks based on multi-sensor fusion. This scheme combines the characteristics and advantages / disadvantages of three different sensors—LiDAR, cameras, and millimeter-wave radar—increasing the recognition weight of each sensor in different scenarios. The obstacle fusion information is then output to the traffic control system for target screening and prediction. This scheme uses two methods for target screening and prediction: the first method uses the unmanned vehicle's attitude as a basis, employing multiple polynomial predictions to determine whether a collision with the fused obstacle will occur within the subsequent travel time (TBD). The second method uses the unmanned vehicle's trajectory as a basis, increasing the prediction expansion width to determine whether the fused obstacle is within the unmanned vehicle's drivable area. After both target screening methods, if a collision with a predicted obstacle occurs, the unmanned vehicle's IPB mechanism executes a braking command to promptly stop the obstacle. Figure 2 The specific steps include:

[0059] Step S100: Summarize and judge the obstacle information identified by each sensor. First, determine whether there is smoke in the current environment. If there is smoke, proceed to step S200. If there is no smoke in the environment, proceed to step S300.

[0060] Step S200: Adjust the fusion weight of each sensor in the smoke environment, reduce the weight of camera and lidar in obstacle fusion, increase the weight of millimeter-wave radar in obstacle fusion, and fuse obstacle information to transmit to the planning and control side for target screening.

[0061] Step S300: Perform multi-sensor obstacle information fusion processing, and transmit the fused obstacle information to the planning and control side for target screening processing;

[0062] In step S400, the control side receives the obstacle fusion target information processed by the perception side and waits for logical judgment.

[0063] In step S500, based on the autonomous vehicle's attitude, multiple polynomial predictions are used to logically determine whether a collision with the fused obstacle will occur within the subsequent travel time (TBD). If the target is within the target screening and predicted collision range, a braking command is output to step S700.

[0064] Step S600: Based on the autonomous vehicle's driving trajectory, increase the prediction expansion width and perform a logical judgment to determine whether the fused obstacle is within the autonomous vehicle's drivable area. If the target is within the target screening prediction collision range, output a braking command to step S700;

[0065] In step S700, if the IPB receives a braking command from the control side, it will perform chassis braking to ensure the safety of the unmanned vehicle.

[0066] In optional embodiments of this application, target selection through multi-sensor fusion can be achieved using more precise sensors. However, compared to unmanned fire trucks, the cost and technological sophistication of lidar, cameras, and millimeter-wave radar sensors in this solution are the most cost-effective. Since individual sensors perform differently in various environments, with varying advantages and disadvantages, multi-sensor fusion can significantly increase the environmental tolerance and robustness of unmanned vehicles in practical applications.

[0067] In summary, this invention integrates logic from multiple sensors in different scenarios with varying proportions, generating obstacle information that can be directly used for logical judgment by the control side. By filtering out redundant obstacles and predicting obstacles using two target selection methods, timely obstacle braking of the entire vehicle can be achieved via IPB (Independent Probe). This improves the overall task execution success rate and vehicle safety.

[0068] Please refer to Figure 3 The diagram illustrates a block diagram of a multi-sensor fusion-based target screening system for unmanned fire trucks, as provided in an embodiment of this application. The system may include:

[0069] The summary judgment module is used to summarize and judge the obstacle information identified by LiDAR, camera and millimeter wave radar, and to determine whether there is smoke in the current environment;

[0070] The processing module is used to process obstacle information according to the smoke judgment results of the current environment; when smoke is detected in the environment, the fusion weights of each sensor are adjusted; when no smoke is detected in the environment, the preset weights of each sensor are kept unchanged.

[0071] The risk assessment module is used to assess collision risks using processed obstacle information; the collision risk assessment includes predictive target selection based on the vehicle's attitude and target selection based on the autonomous vehicle's planned path.

[0072] If the braking processing module determines that there is a risk of collision, it executes a braking command through the IPB mechanism to perform braking processing.

[0073] Specific limitations regarding the multi-sensor fusion-based unmanned fire truck target screening system can be found in the limitations of the multi-sensor fusion-based unmanned fire truck target screening method described above, and will not be repeated here. Each module in the aforementioned multi-sensor fusion-based unmanned fire truck target screening system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0074] In one embodiment, an electronic device is provided, which may be a computer, and its internal structure diagram may be as follows: Figure 4 As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for target screening data of unmanned fire trucks based on multi-sensor fusion. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a target screening method for unmanned fire trucks based on multi-sensor fusion.

[0075] Those skilled in the art will understand that, Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0076] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described target screening method for unmanned fire trucks based on multi-sensor fusion.

[0077] In one embodiment of this application, a computer program product is provided, including a computer program / instructions, which, when executed by a processor, implements the steps of the above-described target screening method for unmanned fire trucks based on multi-sensor fusion.

[0078] The computer-readable storage medium and computer program product provided in this embodiment are similar in implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0079] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0080] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A target screening method for unmanned fire trucks based on multi-sensor fusion, characterized in that, The method includes: The obstacle information identified by lidar, camera and millimeter-wave radar is aggregated and judged to determine whether there is smoke in the current environment; Obstacle information is processed based on the smoke detection results of the current environment; when smoke is detected in the environment, the fusion weights of each sensor are adjusted; when no smoke is detected in the environment, the preset weights of each sensor remain unchanged. Collision risk assessment is performed using processed obstacle information; this includes target selection based on the vehicle's attitude and target selection based on the autonomous vehicle's planned path. If a collision risk is determined, the IPB mechanism will execute a braking command to bring the vehicle to a stop.

2. The target screening method for unmanned fire trucks according to claim 1, characterized in that, Based on the smoke judgment results of the current environment, when smoke is detected in the environment, the weight of the millimeter-wave radar is adjusted to a preset high weight value, while the weights of the camera and lidar are adjusted to preset low weight values ​​respectively.

3. The target screening method for unmanned fire trucks according to claim 1, characterized in that, Target selection and judgment are performed based on the vehicle's posture, specifically including: Based on the autonomous vehicle's attitude, the vehicle's trajectory is fitted by multiple polynomials to predict its position in the future. Based on the predicted location of the autonomous vehicle and obstacle information, determine whether there is a risk of collision between the autonomous vehicle and the obstacle.

4. The target screening method for unmanned fire trucks according to claim 1, characterized in that, Target selection and judgment are performed based on the autonomous vehicle's planned path, specifically including: Based on the planned path of the autonomous vehicle, determine the area where the autonomous vehicle can drive; The obstacle information is compared with the autonomous vehicle's drivable area to determine whether the obstacle is on the autonomous vehicle's driving path.

5. The target screening method for unmanned fire trucks according to claim 1, characterized in that, The obstacle information identified by LiDAR, cameras, and millimeter-wave radar is aggregated and analyzed, including: The obstacle information identified by lidar, cameras and millimeter-wave radar is converted into a unified data format. The converted obstacle information is then fused to generate comprehensive obstacle information.

6. The target screening method for unmanned fire trucks according to claim 1, characterized in that, If a collision risk is assessed, the IPB mechanism will execute a braking command to bring the vehicle to a stop, specifically including: After determining that there is a risk of collision, a braking command is generated and sent to the IPB mechanism; After receiving the braking command, the IPB mechanism calculates the required deceleration to ensure that the autonomous vehicle stops within a safe distance; The IPB mechanism controls the braking system to apply corresponding braking force based on the calculated deceleration; Monitor the braking status of the unmanned vehicle to confirm that it has successfully stopped.

7. A target screening system for unmanned fire trucks based on multi-sensor fusion, characterized in that, The system includes: The summary judgment module is used to summarize and judge the obstacle information identified by LiDAR, camera and millimeter wave radar, and to determine whether there is smoke in the current environment; The processing module is used to process obstacle information according to the smoke judgment results of the current environment; when smoke is detected in the environment, the fusion weights of each sensor are adjusted; when no smoke is detected in the environment, the preset weights of each sensor are kept unchanged. The risk assessment module is used to assess collision risks using processed obstacle information; the collision risk assessment includes predictive target selection based on the vehicle's attitude and target selection based on the autonomous vehicle's planned path. If the braking processing module determines that there is a risk of collision, it executes a braking command through the IPB mechanism to perform braking processing.

8. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, implements the unmanned fire truck target screening method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the unmanned fire truck target screening method as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the unmanned fire truck target screening method according to any one of claims 1 to 6.

Images