A multi-sensor fusion target detection method and system in dusty weather

Abstract

The present invention discloses a multi-sensor fusion target detection method and system under sandstorm weather, which relates to the field of autonomous driving technology in mining areas, including collecting visible light image features and infrared image features under sandstorm weather for feature fusion; predicting the fusion feature position under sandstorm weather based on a confidence network, and initializing the position encoding of the target query; constructing the mutual relationship of the target query based on the attention mechanism, and updating the target query based on the fusion feature; inputting the target query, predicting the target category and target bounding box of the target query, and dynamically judging the prediction result. The present invention can provide richer semantic information and enhance the accuracy of target detection and recognition through the advantages of combined features; initializing the position encoding of the target query helps to better track the target in the subsequent operation of the model, and enhances the accuracy and robustness; further screening and adjusting the prediction results to improve the performance and safety of the autonomous driving system.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology in mining areas, specifically to a multi-sensor fusion target detection method and system under sandstorm weather conditions. Background Technology

[0002] In recent years, with the development of artificial intelligence technology, autonomous driving technology has matured and been applied in various scenarios such as mining areas, ports, and highway logistics. In mining environments, autonomous driving technology can empower transport vehicles to drive autonomously, thereby improving the efficiency and safety of mining transportation. However, the natural environment in mining areas is harsh, especially with frequent sandstorms, posing a dual challenge to sensor hardware performance and autonomous driving perception algorithms. To cope with the complex and ever-changing harsh environment and improve the robustness of perception, multi-sensor fusion technology has been widely used in autonomous driving perception modules.

[0003] Visible light cameras can perceive rich details and textures, but their image quality is affected by dust, leading to missed detections and impacting the accuracy and stability of autonomous driving. Conversely, infrared light has a longer wavelength than visible light, allowing for stronger atmospheric penetration and clear imaging even in adverse weather conditions. This can compensate for the limitations of visible light cameras and improve perception capabilities in scenarios like dust storms and haze. Therefore, theoretically, fusion perception based on visible light and infrared cameras can improve the robustness of the perception system to dusty weather. A key challenge is how to effectively fuse the complementary information from visible light and infrared images to achieve robust target detection. Existing methods for fusing visible light and infrared images generally employ data-level or feature-level fusion. This involves fusing the visible light and infrared images to obtain a fused image, which is then used for downstream perception tasks. This method processes information fusion and perception tasks separately, with the two models optimized independently during training, failing to achieve end-to-end unified optimization. A more advanced approach involves enhancing features in high-entropy regions and attenuating features in low-entropy regions to achieve adaptive fusion of different modal features, followed by target detection based on the fused features. The drawback of this method is that sensor entropy is an artificially designed metric, which is limited by human cognition and cannot be optimized end-to-end with network parameters. Therefore, this invention proposes a multi-sensor fusion target detection method based on visible light and infrared cameras under sandstorm conditions. This method adaptively fuses two modalities at both the feature and target levels, enabling end-to-end optimization of information fusion and target detection to achieve better target detection results. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is that existing multi-sensor fusion techniques suffer from low robustness, low efficiency, and low accuracy, and that there is a problem of how to effectively fuse complementary information from visible light images and infrared images to achieve robust target detection.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a multi-sensor fusion target detection method under sandstorm weather, comprising: collecting visible light image features and infrared image features under sandstorm weather for feature fusion; predicting the location of fused features under sandstorm weather based on a confidence network, and initializing the location encoding of the target query; constructing the interrelationship of the target query based on an attention mechanism, and updating the target query based on the fused features; inputting the target query, predicting the target category and target bounding box of the target query, and dynamically judging the prediction results.

[0007] As a preferred embodiment of the multi-sensor fusion target detection method under sandstorm weather described in this invention, wherein: the acquisition of visible light image features and infrared image features under sandstorm weather includes extracting features from the visible light image using a visible light feature encoder and extracting features from the infrared image using an infrared image feature encoder, and the magnitudes of the visible light features and infrared features are represented as follows:

[0008] [C, H, W]

[0009] Where C is the number of channels, H is the height, and W is the width; the visible light image features and infrared image features under dusty weather include dust density, dust concentration, road visibility, and temperature differences.

[0010] As a preferred embodiment of the multi-sensor fusion target detection method under sandstorm weather described in this invention, the feature fusion includes superimposing visible light features and infrared features under sandstorm weather in the channel dimension to obtain superimposed features of size [2C, H, W]. The superimposed features are used as input, and attention weights of size [2C, H, W] are predicted through a weight network. The attention weights are then split into visible light feature weights and infrared feature weights of size [C, H, W] under sandstorm weather. Based on the feature weights, the visible light features and infrared features under sandstorm weather are weighted and fused to obtain fused features of size [C, H, W].

[0011] As a preferred embodiment of the multi-sensor fusion target detection method under sandstorm weather described in this invention, the initial target query includes location encoding and content encoding under sandstorm weather. The fusion features under sandstorm weather are input, and a target confidence map of size [H, W] is predicted through a confidence network. The N positions with the highest target confidence are selected to initialize the location encoding of the target query. The content encoding of the target query is calculated based on the fusion features under sandstorm weather. The location encoding and content encoding of each target query are added together to obtain N initialized target queries.

[0012] As a preferred embodiment of the multi-sensor fusion target detection method under sandstorm weather described in this invention, the following steps are included: updating the target query involves using the target query as an information carrier, updating the target query content using a target query update module, interacting with the target queries based on a self-attention layer, and constructing the interrelationships of the target queries; interacting with the infrared and visible light features under sandstorm weather based on a cross-attention layer, and adaptively fusing sandstorm density, sandstorm concentration, road visibility, and temperature differences; given N target queries Q = {q 1, …,q N}, for the target query q i The similarity between the dot product calculation and the target query is used as the fusion weight. Based on the weight, the N target queries are weighted and summed to calculate the target query update residual Δq. i , is represented as:

[0013]

[0014] The target query update residual is added to the input target query to calculate the updated target query. The updated target query is then passed through a cross-attention layer to begin fusing infrared and visible light features under sandstorm conditions. The target query is then transformed based on a feedforward neural network to obtain the updated target query.

[0015] As a preferred embodiment of the multi-sensor fusion target detection method under sandstorm weather described in this invention, wherein: the predicted target query includes N given target queries Q = {q 1, …,q N}, for the target query q i Based on a classification prediction network, the target category is predicted; and based on a bounding box regression network, the target bounding box is predicted. The classification prediction network includes fully connected layers and a softmax layer, and outputs a vector of dimension M, where M is the number of categories. The bounding box regression network includes fully connected layers, and its output vector is represented as follows:

[0016] (x, y, w, h)

[0017] This includes (x, y) representing the target center under dusty weather conditions, and (w, h) representing the target width and height under dusty weather conditions.

[0018] As a preferred embodiment of the multi-sensor fusion target detection method under sandstorm weather described in this invention, the dynamic judgment includes cyclically performing target query updates and predictive target queries, and dynamically judging the prediction results.

[0019] The target query update loop condition is to compare two consecutive prediction results. If the target categories are the same and the intersection and union of the foreground targets is less than 0.5, the update loop continues and no prediction result is output.

[0020] When the target categories are the same and the intersection-union ratio of the foreground targets is greater than 0.5, stop the update loop and directly output the prediction result.

[0021] When the update loop reaches 10 times, stop the update loop and directly output the prediction result.

[0022] The number of iterations is dynamically adjusted based on the dynamic convergence of the prediction results.

[0023] Another objective of this invention is to provide a multi-sensor fusion target detection system for sandstorm weather, which can dynamically judge the prediction results by cyclically updating and predicting target queries, thus solving the problem of low detection accuracy in current multi-sensor fusion technologies.

[0024] As a preferred embodiment of the multi-sensor fusion target detection system under sandstorm weather described in this invention, it includes: a feature acquisition module, a feature encoding module, a target update module, and a dynamic judgment module; the feature acquisition module is used to acquire visible light image features and infrared image features under sandstorm weather and perform feature fusion; the feature encoding module is used to predict the fused feature positions under sandstorm weather based on a confidence network and initialize the position encoding of the target query; the target update module is used to construct the interrelationship of the target query based on an attention mechanism and update the target query based on the fused features; the dynamic judgment module is used to predict the target category of the target query using a classification prediction network, predict the target bounding box using a bounding box regression network, and dynamically judge the prediction results.

[0025] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement a multi-sensor fusion target detection method under sandstorm weather.

[0026] A computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of a multi-sensor fusion target detection method under sandstorm weather.

[0027] The beneficial effects of this invention are as follows: The multi-sensor fusion target detection method for sandstorm weather provided by this invention collects visible light image features and infrared image features under sandstorm weather conditions and performs feature fusion. By leveraging the advantages of combined features, it provides richer semantic information and enhances the accuracy of target detection and recognition. Predicting the location of fused features under sandstorm weather conditions and initializing the location encoding of target queries helps to better track targets in subsequent model operations, enhancing the accuracy and robustness of target detection and tracking. Based on the attention mechanism, it constructs the interrelationships of target queries and updates target queries based on fused features, which can improve the accuracy of target detection and tracking, especially for more accurate target positioning and tracking under sandstorm weather conditions. Using a classification prediction network to predict the target category of the target query and a bounding box regression network to predict the target bounding box, the prediction results are dynamically judged, allowing for further filtering and adjustment of the prediction results, improving the performance and safety of autonomous driving systems in this environment. This invention achieves better results in terms of accuracy, robustness, and detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0029] Figure 1 The first embodiment of the present invention provides an overall flowchart of a multi-sensor fusion target detection method under sandstorm weather.

[0030] Figure 2 This is a schematic diagram of feature fusion for a multi-sensor fusion target detection method under sandstorm weather, provided in the first embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram illustrating the target query initialization of a multi-sensor fusion target detection method under sandstorm weather, provided in the first embodiment of the present invention.

[0032] Figure 4 This is a schematic diagram illustrating the target query update of a multi-sensor fusion target detection method under sandstorm weather, provided in the first embodiment of the present invention.

[0033] Figure 5 The following is an overall flowchart of a multi-sensor fusion target detection system under sandstorm weather, provided as a third embodiment of the present invention. DETAILED DESCRIPTION

[0034] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0035] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0036] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0037] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0038] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0039] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0040] Example 1

[0041] Reference Figures 1-4As an embodiment of the present invention, a multi-sensor fusion target detection method under sandstorm weather is provided, comprising:

[0042] S1: Collect visible light image features and infrared image features under sandstorm weather and perform feature fusion.

[0043] Furthermore, visible light features are extracted using a visible light image feature encoder (ResNet-50); infrared features are extracted using an infrared image feature encoder (ResNet-50); both visible light and infrared features are [C, H, W], where C, H, and W represent the number of channels, height, and width, respectively; the visible light and infrared image features under dusty weather conditions include dust density, dust concentration, road visibility, and temperature differences.

[0044] It should be noted that, referring to Figure 2 This process represents the fusion of visible light and infrared features. Visible light and infrared features are superimposed along the channel dimension to obtain a superimposed feature of size [2C, H, W]. Using the superimposed feature as input, a weighted network (two convolutional layers and one sigmoid layer) predicts attention weights of size [2C, H, W], with weight values ​​ranging from [0, 1]. Then, the attention weights are split into visible light feature weights and infrared feature weights, each of size [C, H, W]. Finally, using the corresponding feature weights, the visible light and infrared features are weighted and fused to obtain a fused feature of size [C, H, W].

[0045] It should also be noted that acquiring visible light image features involves using appropriate optical equipment (such as cameras or drones) to capture visible light images during dusty weather. Using these devices, photographic techniques are combined with image processing to extract visual features such as shape, texture, and color from the images, and multi-scale feature extraction and convolutional neural network feature extraction are performed. Acquiring visible light image features can provide rich visual information, including the appearance, shape, and texture of objects. These features play an important role in target detection and recognition, and can help the system understand the scene under dusty weather more accurately.

[0046] It should also be noted that acquiring infrared image features requires specialized equipment such as infrared thermal imagers; by capturing infrared images during sandstorms, key information such as the thermal energy distribution and temperature differences of objects can be obtained; similar to the processing of visible light image features, useful features can be extracted from infrared images through multi-scale feature extraction and convolutional neural network feature extraction methods; acquiring infrared image features can provide information such as the thermal energy distribution and temperature differences of objects, which is also of great significance during sandstorms; infrared image features can help the system perform target detection and recognition in low visibility conditions, improving the robustness of the system.

[0047] It should also be noted that the fusion of visible light image features and infrared image features can be achieved through a fusion network. This network combines information from both types of features, using operations such as convolution and pooling to combine them. In this process, an attention mechanism is introduced to weight the importance of different features. This allows for better aggregation of information from both types of features, improving the final feature representation capability. Acquiring infrared image features can provide information such as the thermal energy distribution and temperature differences of objects, which is also of great significance in dusty weather. Infrared image features can help the system perform target detection and recognition in low visibility conditions, improving the robustness of the system.

[0048] S2: Based on the confidence network, predict the location of fused features under sandstorm weather and initialize the location encoding of the target query.

[0049] Furthermore, refer to Figure 3 This is represented as the target query initialization process. The target query represents a potential target, which includes two parts: location encoding and content encoding. The input is the fused features. The confidence network (1 convolutional layer and 1 sigmoid layer) predicts a target confidence map of size [H, W]. The confidence of each position is between 0 and 1, which indicates the probability that a target exists at that position.

[0050] It should be noted that the N positions with the highest target confidence are selected to initialize the position encoding of the target query, and the fusion features of the corresponding positions are used as the content encoding of the target query. Finally, the position encoding and content encoding of each target query are added together to obtain N initialized target queries.

[0051] It should also be noted that the location of the image features after feature fusion is predicted using a confidence network during sandstorm weather. Furthermore, initializing the location encoding of the target query using the predicted location information helps the model track the target better in subsequent runs.

[0052] S3: Based on the attention mechanism, construct the interrelationships of the target queries and update the target queries based on the fused features.

[0053] Furthermore, refer to Figure 4 This is represented as the target query update process. This step uses the target query as the information carrier and uses the target query update module to update and optimize the target query. In the target query update module, a self-attention layer is used to interact between target queries and build the interrelationships between them. In addition, a cross-attention layer is used to realize the interaction between the target query and infrared and visible light features, adaptively fusing infrared and visible light features to make full use of the complementary information of infrared and visible light features.

[0054] It should be noted that, firstly, a self-attention layer is used to construct the relationships between target queries; given N target queries Q = {q 1, …,q N}, for any target query q i The dot product is used to calculate the similarity between the target query and all target queries, which is then used as the fusion weight. The N target queries are then weighted and summed based on these weights to obtain the target query update residual Δq. i Expressed as:

[0055]

[0056] Then, the target query update residual is added to the input target query to obtain the updated target query; here, the target query residual Δq i This indicates the current target query q. i New information fused from all other target queries.

[0057] After the target query is updated by the self-attention layer, infrared features and visible light features are then fused sequentially through the cross-attention layer. Taking the fusion of infrared features using the cross-attention layer as an example, given N target queries Q = {q 1, …,q N Infrared feature F ir ∈R C×H×W For any target query q i First, a fully connected network is used to predict nine sampling locations. Then, bilinear sampling is used to extract the corresponding feature vectors at these nine sampling locations of the infrared features, resulting in nine infrared feature vectors. Then, use dot product to calculate the current target query q. i The similarity with these nine infrared feature vectors is used as the fusion weight; then, these nine infrared feature vectors are weighted and fused to obtain the target query update residual Δq. i Calculation reference Then, the target query update residual is added to the input target query to obtain the updated target query; here, the target query residual Δq i This indicates the current target query q. iNew information is fused from infrared features; after the infrared and visible light features are fused sequentially through a cross-attention layer, the target query is further transformed by a feedforward neural network to obtain an updated target query.

[0058] S4: Input the target query, predict the target category and target bounding box of the target query, and dynamically judge the prediction results.

[0059] Furthermore, given N target queries Q = {q} 1, …,}, for any target query q i The system uses a classification prediction network to predict the target category and a bounding box regression network to predict the target bounding box. The classification prediction network consists of two fully connected layers and a Softmax layer, and outputs a vector of dimension M, where M is the number of categories. The bounding box regression network consists of two fully connected layers and outputs a vector of dimension 4 (x, y, w, h), which includes the target center (x, y) and the target width and height (w, h).

[0060] It should be noted that the loop update conditions are defined as follows: if any one of them is met, the loop update will stop and the latest detection result will be output directly; compare the latest detection result with the previous detection result. If all corresponding targets have the same category and the intersection-union ratio of the corresponding foreground targets is greater than 0.5, the loop count will reach 10 times.

[0061] It should also be noted that the minimum number of updates is two, and the maximum is 10. The number of updates is dynamically adjusted based on the dynamic convergence of the prediction results, in order to pursue higher detection accuracy while minimizing computational consumption.

[0062] Example 2

[0063] One embodiment of the present invention provides a multi-sensor fusion target detection method under sandstorm weather. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0064] First, visible light and infrared features under sandstorm conditions are extracted, both with a size of [2, 1, 1]. These features are then superimposed along the channel dimension to obtain a superimposed feature of size [4, 1, 1]. Using this superimposed feature as input, a weighted network predicts attention weights of size [4, 1, 1] with a weight value range of 0.8. Next, the attention weights are split into visible light feature weights and infrared feature weights, both with a size of [2, 1, 1]. Finally, the visible light and infrared features are weighted and fused using the corresponding feature weights to obtain a fused feature of size [1, 1, 1].

[0065] Given N target queries Q = {q 1, …,q N Infrared feature Fir ∈R C×H×W For any target query q i First, a fully connected network is used to predict nine sampling locations. Then, bilinear sampling is used to extract the corresponding feature vectors at these nine sampling locations of the infrared features, resulting in nine infrared feature vectors. Then, use dot product to calculate the current target query q. i The similarity with these nine infrared feature vectors is used as the fusion weight; then, these nine infrared feature vectors are weighted and fused to obtain the target query update residual Δq. i Calculation reference The updated target query residual is added to the input target query to obtain the updated target query; here, the target query residual Δq i This indicates the current target query q. i New information is fused from infrared features; after fusing infrared and visible light features sequentially through a cross-attention layer, the target query is further transformed using a feedforward neural network to obtain an updated target query; inputting the target query, predicting the target category and target bounding box, dynamically judging the prediction results, and obtaining the first detection result R1 after predicting and detecting the target; then, based on the latest target query, re-executing the target query update and predicting and detecting the target to obtain the second detection result R2; based on the loop update condition, it is determined whether to directly output the latest result R2, or continue to loop to update the target query and predict and detect the target to predict a new detection result.

[0066] Table 1 shows a comparison of the experimental results. It can be seen that under the same experimental conditions, even with feature fusion and target query update loops, our invention did not significantly increase the recognition time. On the contrary, the detection accuracy increased from 68% to 91%, indicating that our invention achieved better results.

[0067] Table 1 Comparison of Existing Technology and Our Invention Experiment

[0068]

[0069] Based on the dynamic convergence of the prediction results, the number of iterations is dynamically adjusted to pursue higher detection accuracy while minimizing computational cost.

[0070] Example 3

[0071] Reference Figure 5 As an embodiment of the present invention, a multi-sensor fusion target detection system for sandstorm weather is provided, including a feature acquisition module, a feature encoding module, a target update module, and a dynamic judgment module.

[0072] The feature acquisition module is used to collect visible light image features and infrared image features under sandstorm weather and perform feature fusion; the feature encoding module is used to predict the location of fused features under sandstorm weather based on a confidence network and initialize the location encoding of the target query; the target update module is used to build the mutual relationship of the target query based on an attention mechanism and update the target query based on the fused features; the dynamic judgment module is used to predict the target category of the target query using a classification prediction network, predict the target bounding box using a bounding box regression network, and perform dynamic judgment on the prediction results.

[0073] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0075] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0076] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc. It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0077] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-sensor fusion target detection method in sandstorm weather, characterized in that: include: Collect visible light image features and infrared image features under sandstorm weather for feature fusion; Image features include dust density, dust concentration, road visibility, and temperature differences; The initialization target query includes the location coding and content coding under dusty weather. The fusion features under dusty weather are input, and the target confidence map is obtained through confidence network prediction. The N locations with the highest target confidence are selected to initialize the location coding of the target query. The content coding of the target query is calculated based on the fusion features under dusty weather. The location coding and content coding of each target query are added together to obtain N initialized target queries Q = {q 1, …,q N }; Based on the attention mechanism, the mutual relationship of the target query is constructed, and the target query is updated based on the fused features; For the target query q i , the similarity between the dot product calculation and the target query is used as the fusion weight, and the N target queries are weighted and summed based on the weight to calculate the target query update residual △q i , expressed as: Add the target query update residual to the input target query to calculate the updated target query; The updated target query begins to fuse the infrared features and visible light features under dust weather through the cross attention layer, and transforms the target query based on the feedforward neural network to obtain an updated target query. Input a target query, predict the target category and target bounding box of the target query, and dynamically judge the prediction results; Dynamic judgment includes cyclically updating target query and predicting target query, and dynamically judging the prediction results.

2. The multi-sensor fusion target detection method in sandstorm weather as claimed in claim 1, characterized in that: The collecting of visible light image features and infrared image features under sandstorm weather includes using a visible light feature encoder to extract features of the visible light image, and using an infrared image feature encoder to extract features of the infrared image. The sizes of the visible light features and the infrared features are expressed as: [C, H, W] Where C is the number of channels, H is the height, and W is the width; The visible light image features and infrared image features in dust weather include dust density, dust concentration, road visibility and temperature difference.

3. The multi-sensor fusion target detection method in sandstorm weather as claimed in claim 2, characterized in that: The feature fusion includes superimposing visible light features and infrared features under sandstorm weather in the channel dimension to obtain superimposed features of a size of [2C, H, W], using the superimposed features as input, predicting attention weights of a size of [2C, H, W] through a weight network, splitting the attention weights into visible light feature weights and infrared feature weights under sandstorm weather, both of which have a size of [C, H, W], and performing weighted fusion of the visible light features and infrared features under sandstorm weather based on the feature weights to obtain fused features of a size of [C, H, W].

4. The multi-sensor fusion target detection method in sandstorm weather as claimed in claim 2, characterized in that: The predicted target query includes given N target queries Q={q 1, …,q N }, for the target query q i , based on the classification prediction network, predict the target category, and predict the target bounding box through the bounding box regression network; The classification prediction network includes a fully connected layer and Softmax, and the output dimension is a vector of M, where M is the number of categories; The bounding box regression network includes a fully connected layer, and the output vector is represented as: (x, y, w, h) Among them, (x, y) is the center of the target under dust weather, and (w, h) is the width and height of the target under dust weather.

5. The multi-sensor fusion target detection method in sandstorm weather as claimed in claim 4, characterized in that: The target query update loop condition is to compare two consecutive prediction results. When the categories of the targets are the same and the intersection and union of the foreground targets is less than 0.5, the update loop continues without outputting the prediction results. When the categories of the targets are the same and the intersection-over-union ratio of the foreground targets is greater than 0.5, the update loop is stopped and the prediction results are directly output; When the update cycle number reaches 10, stop the update cycle and directly output the prediction result; According to the dynamic convergence of the prediction results, the number of loop updates is dynamically adjusted.

6. A system using the multi-sensor fusion target detection method in sandstorm weather as claimed in any one of claims 1 to 5, characterized in that: It includes feature collection module, feature encoding module, target update module and dynamic judgment module; The feature acquisition module is used to collect visible light image features and infrared image features in sandstorm weather for feature fusion; The feature encoding module is used to predict the fusion feature position under sandstorm weather based on the confidence network and initialize the position encoding of the target query; The target update module is used to build the relationship between target queries based on the attention mechanism and update the target query based on the fusion feature; The dynamic judgment module is used to use a classification prediction network to predict the target category of the target query, use a bounding box regression network to predict the target bounding box, and dynamically judge the prediction results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the multi-sensor fusion target detection method in sandstorm weather described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-sensor fusion target detection method in sandstorm weather described in any one of claims 1 to 5 are implemented.

Images