Target detection method and device in severe weather, electronic equipment and storage medium

Abstract

The invention provides a target detection method and device in severe weather, electronic equipment and a storage medium, and the method comprises the steps: obtaining a scene image of a target region, carrying out the recognition of the scene image through a target detection model, and obtaining a multispectral target detection result of the target region, wherein the target detection model is obtained by training a CME-YOLO framework and a CPTFusion framework. The method has the beneficial effect that the accuracy of target detection in the severe weather is improved.

Description

Technical Field

[0001] The present invention relates to the fields of image detection and computer technology, and particularly to a method, device, electronic device and storage medium for target detection in bad weather. Background Art

[0002] The multi-spectral input combining visible light and thermal infrared images can provide richer complementary information for object detection. Nevertheless, the existing visible light-infrared object detection methods are not well applied to object detection under bad weather conditions. Due to the particularity of environmental disturbances such as noise and occlusion in rainy, foggy and snowy weather, the existing methods still choose to adopt corresponding processing algorithms for special weather and cannot perform unified processing. And because the interference situations of visible light are different, the existing technologies cannot effectively detect targets for different bad weather conditions. Summary of the Invention

[0003] The main purpose of the embodiments of the present invention is to propose a method, device, electronic device and storage medium for target detection in bad weather, which improves the accuracy of target detection in bad weather.

[0004] One aspect of the present invention provides a method for target detection in bad weather, including:

[0005] Obtain a scene image of a target area, and use a target detection model to identify the scene image to obtain a multi-spectral target detection result of the target area;

[0006] The training process of the target detection model includes:

[0007] Obtain clean images and real-scene bad weather images under different illuminations, synthesize the clean images to obtain a first synthesized bad weather image, and combine the real-scene bad weather image and the first synthesized bad weather image to obtain a combined image;

[0008] Use CycleGan to input the real-scene image and the combined image as training samples, and obtain a second synthesized bad weather image paired with the clean image through training;

[0009] Combine the second synthesized bad weather image and the real-scene image into a training data set, where the training data set includes visible light images and infrared images;

[0010] Use the CME-YOLO framework to perform upsampling processing on the training data set to obtain a sampling result, perform multi-modal fusion on the sampling result through CPTFusion, obtain a fusion feature, and detect the fusion feature to obtain a detection result, and complete the training of the target detection model according to the detection result.

[0011] According to the target detection method in bad weather, the synthesis of the clean image to obtain the first synthetic bad weather image includes:

[0012] Using a particle system and a shader to add bad weather effects to the clean image, where the bad weather effects include raindrops, snow, and fog;

[0013] Among them, the simulation formulas for the bad weather effects of raindrops and snow are

[0014] RainEffect = function(tiltAngle, rainSIze, rainSpeed)

[0015] where tiltAngle controls the tilt angle of raindrops or snow, rainSize represents the size of raindrops or snow, and rainSpeed controls the falling speed of raindrops or snow;

[0016] Among them, the simulation of the bad weather effect of fog is through an atmospheric scattering model, and the formula of the atmospheric scattering model is:

[0017] I(x) = I0(x)·e -β·d(x) +A·(1 - e -β·d(x) )

[0018] where I0(x) is the pixel intensity without fog, β is the atmospheric scattering coefficient, d(x) is the distance from the pixel point to the camera, A is the atmospheric light intensity, and β and A are used to adjust the concentration and brightness of the fog.

[0019] According to the target detection method in bad weather, where CycleGan is used to input real scene images and the combined images as training samples, and a second synthetic bad weather image paired with the clean image is obtained through training, including:

[0020] When training the training samples through the CycleGan, a generator and a discriminator are used for domain conversion, and the training is evaluated through the loss function of the CycleGan;

[0021] Adjust the parameters during the training of the generator to control the intensity and features during the generation of the second synthetic bad weather image.

[0022] According to the target detection method in bad weather, where the loss function of CycleGan includes adversarial loss, cycle consistency loss, and identity mapping loss.

[0023] According to the target detection method in bad weather, where the CME-YOLO framework is used to perform upsampling processing on the training dataset to obtain a sampling result, including:

[0024] Interpolate new data points for the original data points of the training data set using bilinear interpolation;

[0025] Use learnable convolution to find the best sampling region according to the data characteristics of the training data set, and perform sampling according to the best sampling region to obtain the sampling result.

[0026] According to the target detection method under the bad weather, wherein the sampling result is subjected to multi-modal fusion through CPTFusion, and the fused feature is obtained, including:

[0027] Reduce the dimensionality of the feature tensors of the visible light image and the infrared image in the sampling result through the CME-YOLO framework and reshape them to obtain the embedding;

[0028] Stitch the embeddings to obtain a unified tensor, and use the residual bottleneck structure and the fork fusion module of the CME-YOLO framework to perform feature fusion and detail expression of the visible light image and the infrared image, and use the Transformer variant module to perform feature enhancement and complex relationship capture processing on the feature fusion of the visible light image and the infrared image through the self-attention mechanism and the multi-layer perceptron to obtain the fused feature.

[0029] According to the target detection method under the bad weather, wherein the method further includes:

[0030] Extract the convolutional features of the visible light image and the infrared image through the intermediate layer of the network of the CME-YOLO framework, convert the convolutional features into sequences and rearrange them to obtain the visible light sequence and the infrared sequence;

[0031] Combine the visible light sequence, the infrared sequence and the learnable position encoding to obtain the input sequence, wherein the input sequence passes through the difference bottleneck structure, the fork fusion module and the Transformer variant module in sequence;

[0032] Process the input sequence through two convolutional layers of different sizes by the residual bottleneck structure to first reduce the dimensionality and then increase the dimensionality;

[0033] Group the input sequence through the fork fusion module and perform convolutional processing to obtain local features and group-intercepted global features, and at the same time use the self-attention mechanism to strengthen the interaction between the local features and the group-intercepted global features;

[0034] Use the three linear transformations of the self-attention mechanism in the Transformer variant module to calculate the query, key and value for the input sequence, calculate the attention weights according to the query, key and value, and perform processing through the linear transformation and activation function of the multi-layer perceptron.

[0035] Another aspect of the embodiments of the present invention provides a target detection device under bad weather, including:

[0036] The first module is configured to obtain a scene image of a target area, identify the scene image by using a target detection model, and obtain a multi-spectral target detection result of the target area;

[0037] The training process of the target detection model includes:

[0038] The second module is configured to obtain clean images and real-scene bad weather images under different illuminations, synthesize the clean images to obtain a first synthesized bad weather image, and combine the real-scene bad weather image and the first synthesized bad weather image to obtain a combined image;

[0039] The third module is configured to use CycleGan to input the real-scene image and the combined image as training samples, and obtain a second synthesized bad weather image paired with the clean image through training;

[0040] The fourth module is configured to combine the second synthesized bad weather image and the real-scene image into a training data set, where the training data set includes visible light images and infrared images;

[0041] The fifth module is configured to perform upsampling processing on the training data set by using the CME-YOLO framework to obtain a sampling result, perform multi-modal fusion on the sampling result through CPTFusion, obtain a fusion feature, and detect the fusion feature to obtain a detection result, and complete the training of the target detection model according to the detection result.

[0042] Another aspect of the embodiments of the present invention provides an electronic device, including a processor and a memory;

[0043] The memory is used to store a program;

[0044] The processor executes the program to implement the method described above.

[0045] The embodiments of the present invention also disclose a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described above.

[0046] The beneficial effects of the present invention are as follows: By constructing a new AWID dataset, it aims to cover various adverse weather scenarios by mathematically formalizing the impact of various weather phenomena on images. Since the current algorithms for solving adverse weather mainly start from image restoration and cannot uniformly process different weather phenomena, the multi-spectral fusion framework CME-YOLO is adopted to unify the methods for image processing and object detection under adverse weather conditions. That is, the framework of YOLOv10 is extended to a two-stream input, which respectively inputs visible light and thermal infrared images, and combines the fusion module CPTFusion and the upsampling module for feature fusion enhancement, thereby improving the accuracy of object detection in adverse weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where:

[0048] Figure 1 is a schematic diagram of the object detection process in adverse weather according to an embodiment of the present invention.

[0049] Figure 2 is a schematic diagram of the process of generating adverse weather by CycleGan according to an embodiment of the present invention.

[0050] Figure 3 is a schematic diagram of the dataset made by CycleGAN according to an embodiment of the present invention.

[0051] Figure 4 is a flowchart of the upsampling process according to an embodiment of the present invention.

[0052] Figure 5 is a flowchart of the multi-modal fusion process according to an embodiment of the present invention.

[0053] Figure 6 is a schematic diagram of the processing of the CME-YOLO framework according to an embodiment of the present invention.

[0054] Figure 7 is a schematic diagram of bilinear interpolation according to an embodiment of the present invention.

[0055] Figure 8 is a schematic diagram of the optimization of the learnable convolutional sampling domain according to an embodiment of the present invention.

[0056] Figure 9 is a schematic diagram of the processing flow of CPTFusion according to an embodiment of the present invention.

[0057] Figure 10 is a schematic diagram of the CPTFusion framework according to an embodiment of the present invention.

[0058] Figure 11 is a schematic diagram of the residual bottleneck structure according to an embodiment of the present invention.

[0059] Figure 12 It is a schematic diagram of the forking and fusion module according to an embodiment of the present invention.

[0060] Figure 13 It is a schematic diagram of the target detection device under bad weather according to an embodiment of the present invention. Specific embodiments

[0061] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. In the following description, suffixes such as "module", "component" or "unit" used to denote elements are only for the convenience of describing the present invention and have no specific meaning in themselves. Therefore, "module", "component" or "unit" can be used interchangeably. "First", "second", etc. are only used to distinguish technical features and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence of the indicated technical features. In the following description, the consecutive numbering of method steps is for the convenience of review and understanding. Considering the overall technical solution of the present invention and the logical relationship between each step, adjusting the execution order between steps will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0062] Figure 1 It is a schematic diagram of the target detection process under bad weather according to an embodiment of the present invention. It includes S100 to S500:

[0063] S100, obtain the scene image of the target area, and identify the scene image using the target detection model to obtain the multi-spectral target detection result of the target area.

[0064] The training process of the target detection model includes S200 to S500:

[0065] S200, obtain clean images and real-scene bad weather images under different illuminations, synthesize the clean images to obtain the first synthesized bad weather image, and combine the real-scene bad weather image and the first synthesized bad weather image to obtain a combined image.

[0066] In some embodiments, the AWID dataset aims to cover various bad weather scenarios by mathematically formalizing the impact of various weather phenomena on images. Specifically, AWID includes 20,000 pairs of aligned visible-infrared image pairs, spanning three weather conditions: rain, fog, snow, and two scenes: day and night, with each condition and scene evenly distributed. Since the current algorithms for solving bad weather mainly start from image restoration, different weather phenomena cannot be uniformly processed.

[0067] In some embodiments, the AWID dataset is obtained as follows:

[0068] The SMOD dataset is selected as the base dataset and extended to the Adverse Weather and Illumination Dataset (AWID). Both day and night scenes come from the real data in SMOD, and rain, snow, and fog are produced according to the embodiments of the present invention.

[0069] The SMOD dataset (SJTU Multispectral Object Detection Dataset) was released by a research team at Shanghai Jiao Tong University. This dataset focuses on object detection in multi-spectral images, especially under complex lighting conditions, aiming to improve the performance of object detection. The SMOD dataset contains 5,378 pairs of day scene images and 3,298 pairs of night scene images. These image pairs are strictly aligned in time and space, covering a rich range of lighting variations, especially in night scenes, including very low illumination and strong light conditions.

[0070] In the embodiments of the present invention, paired rain, fog, and snow images are synthesized on the clean day images of SMOD, and then real rain, fog, and snow images in outdoor scenes are collected. Together with the day scene data in SMOD, they are used as training samples and input into CycleGan to obtain the weights for rain, fog, and snow image translation respectively, so as to generate adverse weather images that are fully paired with the day scene data in SMOD.

[0071] To ensure the high alignment of pixels and prevent the threat of pixel movement caused by unpaired training data in CycleGan training data. In the embodiments of the present invention, the data generated by CycleGan for the first time is used as training samples to cyclically train CycleGan until stable day-adverse weather paired images are generated. The generated dataset and the original SMOD data are respectively input into Yolov10 for object detection tasks to verify the effectiveness of dataset generation.

[0072] As shown in Table 1, under the influence of adverse weather, the detection accuracy decreases. Within the normal range of weather influence, the embodiments of the present invention consider that the pixel movement problem caused by image translation can be ignored in the object detection evaluation task.

[0073] Table 1 Detection results of synthetic weather images and original weather images (day and night) by YOLOv10

[0074] Day Night Hazy Rain Snow Yolov10 69.29 64.87 67.91 64.43 67.52

[0075] In some embodiments, for adding rain and snow effects, the embodiments of the present invention adopt the method of particle system. The particle system is a technology for simulating complex phenomena. It simulates natural phenomena such as raindrops and snowflakes by generating a large number of small particles. In this study, the embodiments of the present invention define parameters such as the size, speed, and tilt angle of the particles, and write the corresponding fragment shader through the Shader language to achieve the visual effects of rain and snow. For example, the simulation of raindrops can be carried out through the following formula:

[0076] RainEffect=function(tiltAngle,rainSIze,rainSpeed)

[0077] Among them, tiltAngle controls the tilt angle of the raindrops, rainSize controls the size of the raindrops, and rainSpeed controls the falling speed of the raindrops. By adjusting these parameters, the embodiments of the present invention can simulate realistic raindrop effects on the image.

[0078] For adding fog effects, the embodiments of the present invention adopt the atmospheric scattering model. This model is based on the principle of light scattering and absorption in the atmosphere, and simulates the fog effect by calculating the scattered light intensity of each pixel point in the image. The simulation of the fog effect can be carried out through the following formula:

[0079] I(x)=I0(x)·e -β·d(x) +A·(1-e -β·d(x) )

[0080] I0(x) is the pixel intensity without fog, β is the atmospheric scattering coefficient, d(x) is the distance from the pixel point to the camera, and A is the atmospheric light intensity. By adjusting the values of β and A, the embodiments of the present invention can control the concentration and brightness of the fog, so as to add different degrees of fog effects to the image.

[0081] S300, use CycleGan to input real scene images and combined images as training samples, and obtain the second synthetic bad weather image paired with the clean image through training.

[0082] In some embodiments, referring to Figure 2 the schematic diagram of the CycleGan generation of bad weather process shown, which includes but is not limited to steps S310~S320:

[0083] S310, when performing training on the training samples through CycleGan, use the generator and discriminator for domain conversion, and evaluate the training through the loss function of CycleGan;

[0084] S320. Adjust the parameters during generation and training to control the intensity and characteristics when generating the second synthetic bad weather images.

[0085] In some embodiments, the CycleGAN generates a dataset

[0086] CycleGAN consists of two pairs of generators and discriminators: G and D Y , and F and D X . Among them, G is used to convert images from domain X (such as sunny day images) to domain Y (such as rainy day images), while F performs the opposite conversion. D Y and D X are discriminators for the corresponding domains, which are used to distinguish real images and images generated by the generator.

[0087] The training of CycleGAN relies on three main loss functions:

[0088] Adversarial Loss:

[0089]

[0090] This loss function encourages generator G to generate images that look like real domain Y images, while encouraging discriminator D Y to distinguish real and generated images.

[0091] Cycle Consistency Loss:

[0092]

[0093] This loss ensures that information during the conversion process is not lost, that is, converting from one domain to another and then back should be close to the original image.

[0094] Identity Loss:

[0095]

[0096] This loss encourages the generator to retain certain features of the image when converting the image, even if the image is already in the target domain.

[0097] To generate a bad weather dataset, the embodiments of the present invention use sunny day images in the SMOD dataset as domain X, and real rain, snow, and fog images collected and synthetic weather images made through special effects as domain Y, and use CycleGAN to convert these images from X to Y. By adjusting the parameters of generator G and the training process, the intensity and characteristics of the generated bad weather effects are controlled.

[0098] During the training process, the parameters of the generator and discriminator are continuously adjusted to minimize the above loss function. In this way, the generator learns how to convert sunny-day images into images with rain, snow, or fog effects without relying on paired samples.

[0099] To prevent the pixel shift caused by the unpaired training data of real weather and SMOD sunny-day images from affecting the accuracy of downstream tasks, the embodiment of the present invention uses the dataset produced by the first training of CycleGAN as training samples to cycle-train CycleGAN until stable paired bad-weather images are generated. Part of the dataset produced by the embodiment of the present invention is as Figure 3 shown.

[0100] S400, combine the second synthetic bad-weather image and the real-scene image into a training dataset, where the training dataset includes visible-light images and infrared images.

[0101] S500, perform upsampling on the training dataset using the CME-YOLO framework to obtain a sampling result, perform multi-modal fusion on the sampling result through CPTFusion, obtain a fusion feature, and detect the fusion feature to obtain a detection result, and complete the training of the object detection model according to the detection result.

[0102] In some embodiments, referring to the Figure 4 upsampling flowchart shown, it includes but is not limited to steps S510 to S520:

[0103] S510, insert new data points into the original data points of the training dataset using bilinear interpolation;

[0104] S520, use a learnable convolution to find the best sampling region according to the data characteristics of the training dataset, and perform sampling according to the best sampling region to obtain a sampling result.

[0105] In some embodiments, referring to the Figure 5 multi-modal fusion flowchart shown, it includes but is not limited to steps S530 to S540:

[0106] S530, reduce the dimensionality of the feature tensors of the visible-light image and the infrared image in the sampling result through the CME-YOLO framework and reshape them to obtain an embedding;

[0107] S540, splice the embeddings to obtain a unified tensor, use the residual bottleneck structure and the fork fusion module of the CME-YOLO framework to perform feature fusion and detail expression of the visible-light image and the infrared image, and use the Transformer variant module to perform feature enhancement and complex relationship capture processing on the feature fusion of the visible-light image and the infrared image through the self-attention mechanism and the multi-layer perceptron to obtain a fusion feature.

[0108] In some embodiments, refer to Figure 6 the schematic diagram of the CME-YOLO framework processing shown.

[0109] The Cross-Perception Transformer Fusion (CME-YOLO) proposed in the embodiments of the present invention includes two stages: feature extraction and detection. The embodiments of the present invention expand the framework of yolov10, combine the multi-modal fusion module and the upsampling module proposed in the embodiments of the present invention, and retain more detailed feature information by means of visible light and infrared images to achieve multi-spectral target detection.

[0110] The Adaptive Upsampling (AdSample) module, the need for upsampling, that is, inserting new data points between the original data points to increase the density of the data. These new data points are calculated by mathematical methods with the aim of making the data look smoother or more detailed without changing the original data trend.

[0111] Bilinear interpolation. Map the x coordinates of the feature map of C*sH*sW back to the feature map of CHW with x / s, y / s, and perform bilinear interpolation with the pixel values and position coordinates of the nearest four points, as Figure 4 shown. At this time, the pixel value of each position obtained by upsampling is closely related to the nearest four points that are mapped back to the original feature map, but the selection of the four points is only related to the point coordinates of x and the upsampling scale factor. The embodiments of the present invention represent upsampling as: X = upsample(x, s).

[0112] Refer to Figure 7 the schematic diagram of bilinear interpolation shown, where x represents the feature map of size CHW, X represents the feature map of CsHsW after upsampling by s, and s represents the upsampling scale factor.

[0113] Refer to Figure 8 the schematic diagram of the learnable convolution sampling domain optimization shown. According to the idea of bilinear interpolation, the effect of upsampling is closely related to the selection of the sampling area. In order to find the best sampling domain, the embodiments of the present invention introduce a learnable convolution, which is represented as Conv() in the formula and acts on the sampling position x to generate the best displacement distance z compared with the original sampling position, as Figure 5 shown, then the sampling domain will become Z = Conv(x), S = s + Z.

[0114] By introducing a learnable convolutional shift to find the optimal sampling domain, the embodiments of the present invention transform the upsampling process into an adaptive learning mechanism. This method is no longer limited to static, predefined mathematical models, but allows the network to dynamically adjust its sampling strategy according to the characteristics of the input data. This adaptive upsampling method enables the network to better capture and utilize the key information in the image, thereby improving the accuracy and efficiency of image processing and analysis.

[0115] In some embodiments, referring to Figure 9 the schematic diagram of the CPTFusion processing flow shown, which includes but is not limited to steps S610 to S650:

[0116] S610, extract the convolutional features of the visible light image and the infrared image through the intermediate layer of the CME - YOLO framework, convert the convolutional features into sequences and rearrange them to obtain a visible light sequence and an infrared sequence;

[0117] S620, combine the visible light sequence, the infrared sequence and the learnable position encoding to obtain an input sequence, and the input sequence sequentially passes through a difference bottleneck structure, a bifurcated fusion module and a Transformer variant module;

[0118] S630, perform a process of first reducing the dimension and then increasing the dimension on the input sequence through two convolutional layers of different sizes by means of a residual bottleneck structure;

[0119] S640, group the input sequence through a bifurcated fusion module and perform convolutional processing to obtain local features and globally capture inter - group features, and at the same time adopt a self - attention mechanism to strengthen the interaction between the local features and the globally captured inter - group features;

[0120] S650, use the three linear transformations of the self - attention mechanism in the Transformer variant module to calculate the query, key and value for the input sequence, calculate the attention weights according to the query, key and value, and perform processing through the linear transformation and activation function of a multi - layer perceptron.

[0121] In some embodiments, as Figure 10 shown in the schematic diagram of the CPTFusion framework, the input infrared and visible light feature tensors first undergo a dimension reduction process and are reshaped into embeddings, and then are spliced to form a unified tensor representation. The fusion and detailed expression of the visible light and infrared features are strengthened through a residual bottleneck structure and a Bifurcated Fusion Module. In addition, the embodiments of the present invention apply a variant Block of the transformer to enhance the complementarity between the infrared and visible light features, and the three parts will be described separately later. In this way, the network can focus on capturing the detailed feature expressions of the two source images while enhancing their complementarity.

[0122] In the embodiments of the present invention, the convolutional features of the RGB image and the infrared image are extracted from the middle layer of the network, denoted as F RGB and F IR , and their dimensions are all B×C×H×W, where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map respectively. First, in the embodiments of the present invention, these feature maps are converted into a sequence form, that is, they are flattened and rearranged to form two sequences I RGB and I IR , and the dimension of each sequence is BHW×C. Then, in the embodiments of the present invention, these two sequences are merged, and a set of learnable position encodings are introduced. These encodings are a set of parameters with a dimension of 2HW×C, which are used to enhance the model's ability to recognize different feature position information. In this way, in the embodiments of the present invention, the input sequence I with a dimension of 2HW×C is obtained. Through the position encoding, the model can distinguish the spatial relationship between different features during the training process. In the embodiments of the present invention, the sequence I is passed through BRM, BFM, and AMPM in sequence to achieve feature fusion and enhancement, and then enhanced through a residual connection. Finally, I is reshaped back to the B×C×H×W input detection head.

[0123] Figure 11 The residual bottleneck structure Residual Bottleneck Structure shown. The sequence I is input into the residual bottleneck structure, which includes two 1×1 convolutional layers and a 3×3 convolutional layer. These convolutional layers are surrounded by batch normalization (BatchNorm) layers, and non-linearity is introduced through the ReLU activation function. Finally, it is enhanced through a residual connection.

[0124] The bottleneck design of this structure reduces the number of parameters of the model by first reducing the dimension and then increasing the dimension. Compared with directly using a 3×3 convolutional layer, the residual bottleneck structure significantly reduces the parameters and computational complexity without losing too much information. Among them, the 1×1 convolutional layer not only reduces or increases the number of channels, but also can perform feature fusion without changing the spatial dimension of the feature map. This fusion can capture cross-channel correlations, thereby generating richer and more abstract feature representations. Due to the reduction in the number of parameters, the bottleneck design also improves the computational efficiency. This enables the network to process information faster, while reducing the memory footprint, making it possible to train deeper or wider networks with limited computational resources. Although the residual bottleneck structure reduces the number of channels in the middle layer, due to the use of non-linear activation functions and residual connections, it can effectively retain important feature information. The residual connection ensures that information can flow directly even when the network is very deep, reducing the problem of gradient disappearance.

[0125] As shown in Figure 10The shown Bifurcated Fusion Module is a key challenge in the field of image processing, such as in fusion tasks, on how to effectively combine global and local features. Although traditional convolutional neural networks can capture local features, they have limitations in dealing with global dependencies.

[0126] The Bifurcated Fusion Module groups the input feature sequence and applies convolutional operations to capture local features within groups and global features between groups. At the same time, it uses the self-attention mechanism to strengthen the interaction between global features and local features. As Figure 12 shown in the schematic diagram of the bifurcated fusion module, where the local path uses a standard 3×3 convolutional kernel to capture detailed information, while the global path extracts more extensive context features through a 1×1 convolutional kernel. This design enables the model to enhance the understanding of the global structure of the image while maintaining computational efficiency and retain important local detail information. The BFM module can flexibly adapt to different feature fusion requirements and effectively improve the model's ability to capture multi-scale features by processing local and global information in parallel.

[0127] The Transformer variant module combines the self-attention mechanism and a multi-layer perceptron to enhance the expressive power of features and capture complex relationships in sequential data, as Figure 10 shown in the Transformer variant module (block).

[0128] The self-attention mechanism allows the model to dynamically focus on different parts of the input sequence globally. For the input sequence I ∈ 2HW×C, the self-attention layer first calculates the query (Q), key (K), and value (V) through three linear transformations:

[0129] Q = IW Q , K = IW K , V = IW V

[0130] where, W Q , W K , W V are learnable weight matrices. Then, the attention weights are calculated:

[0131]

[0132] where, d k is the dimension of the key (K), used to scale the dot product to stabilize the training process. The attention weights represent the influence degree of each element in the input sequence on other elements.

[0133] The multi-layer perceptron, the output of the self-attention layer is then passed through an MLP layer, which consists of two linear transformations and an activation function, used to introduce non-linearity:

[0134] MLP(I) = max(0, IW1 + b1)W2 + b2

[0135] where W1, b1, W2, b2 are learnable parameters. The MLP layer enhances the expressive power of the model, allowing it to capture more complex feature relationships.

[0136] Figure 13 It is a diagram of the target detection and analysis device in the embodiments of the present invention. The device includes a first module 1310, a second module 1320, a third module 1330, a fourth module 1340 and a fifth module 1350.

[0137] Among them, the first module is used to obtain the scene image of the target area, identify the scene image using the target detection model, and obtain the multi-spectral target detection result of the target area; the training process of the target detection model includes: the second module is used to obtain clean images and real-scene bad weather images under different illuminations, synthesize the clean images to obtain the first synthesized bad weather image, and combine the real-scene bad weather image and the first synthesized bad weather image to obtain a combined image; the third module is used to use CycleGan to input the real-scene image and the combined image as training samples, and obtain the second synthesized bad weather image paired with the clean image through training; the fourth module is used to combine the second synthesized bad weather image and the real-scene image into a training data set, where the training data set includes visible light images and infrared images; the fifth module is used to perform upsampling processing on the training data set using the CME-YOLO framework to obtain a sampling result, perform multi-modal fusion on the sampling result through CPTFusion, obtain a fusion feature, and detect the fusion feature to obtain a detection result, and complete the training of the target detection model according to the detection result.

[0138] Exemplarily, with the cooperation of the first to fifth modules in the device, the example device can implement any of the foregoing target detection methods in bad weather, that is, obtain a scene image of the target area, use a target detection model to identify the scene image, and obtain a multi-spectral target detection result of the target area; S200, obtain clean images and real-scene bad weather images under different illuminations, synthesize the clean images to obtain a first synthesized bad weather image, and combine the real-scene bad weather image and the first synthesized bad weather image to obtain a combined image; S300, use CycleGan to input the real-scene image and the combined image as training samples, and obtain a second synthesized bad weather image paired with the clean image through training; S400, combine the second synthesized bad weather image and the real-scene image into a training data set, where the training data set includes visible light images and infrared images; S500, perform upsampling processing on the training data set using the CME-YOLO framework to obtain a sampling result, perform multi-modal fusion on the sampling result through CPTFusion, obtain a fusion feature, and detect the fusion feature to obtain a detection result, and complete the training of the target detection model according to the detection result. The beneficial effects of the present invention are as follows: By constructing a new AWID data set, it aims to cover various bad weather scenarios by mathematically formalizing the influence of various weather phenomena on images. Since the current algorithms for solving bad weather mainly start from image restoration and cannot uniformly process different weather phenomena; adopting the multi-spectral fusion framework CME-YOLO unifies the way of processing image processing and target detection under bad weather conditions, that is, extends the framework of YOLOv10 to dual-stream input, inputs visible light and thermal infrared images respectively, combines the fusion module CPTFusion and the upsampling module, performs feature fusion enhancement, and improves the accuracy of target detection under bad weather.

[0139] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory;

[0140] The memory stores a program;

[0141] The processor executes the program to execute the foregoing target detection method in bad weather; this electronic device has the function of carrying and running the software system for target detection in bad weather provided by the embodiment of the present invention. For example, a personal computer, a minicomputer, a mainframe, a workstation, a network or a distributed computing environment, a separate or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0142] An embodiment of the present invention further provides a computer-readable storage medium, the storage medium stores a program, and the program is executed by a processor to implement the target detection method in bad weather as described above.

[0143] In some alternative embodiments, the functions / operations recited in the block diagrams may not occur in the order shown in the operational illustrations. For example, depending on the functions / operations involved, two blocks shown in succession may actually be executed substantially simultaneously or the blocks may sometimes be executed in the reverse order. Further, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and in which sub-operations described as part of a larger operation are performed independently.

[0144] An embodiment of the present invention also discloses a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device may read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the foregoing target detection method under bad weather conditions.

[0145] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features described may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the module will be understood within the ordinary skill of the engineer, given the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein. Accordingly, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It should also be understood that the particular concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0146] When the above-mentioned functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0148] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or more wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or, if necessary, other suitable processing, and then stored in a computer memory.

[0149] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0151] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0152] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for detecting a target in bad weather, characterized in that: include:. Acquire a scene image of a target area, and use a target detection model to identify the scene image to obtain a multispectral target detection result of the target area; The training process of the target detection model includes: Acquire clean images and real scene severe weather images under different illuminations, synthesize the clean images to obtain a first synthesized severe weather image, and combine the real scene severe weather image and the first synthesized severe weather image to obtain a combined image; CycleGan is used to input a real scene image and the combined image as training samples, and a second synthetic severe weather image paired with the clean image is obtained through training; combining the second synthetic severe weather image and the real scene image into a training data set, wherein the training data set includes a visible light image and an infrared image; The CME-YOLO framework is used to perform upsampling processing on the training data set to obtain sampling results, and multimodal fusion is performed on the sampling results through CPTFusion to obtain fusion features, and the fusion features are detected to obtain detection results, and the training of the target detection model is completed according to the detection results.

2. The method for detecting targets in bad weather according to claim 1, characterized in that: The step of synthesizing the clean image to obtain a first synthesized severe weather image includes: Adding severe weather effects to the clean image using a particle system and a shader, wherein the severe weather effects include raindrops, snow, and fog; The simulation formula for the severe weather effects of raindrops and snow is: RainEffect = function (tiltAngle, rainSize, rainSpeed), where tiltAngle controls the tilt angle of raindrops or snow, rainSize controls the size of raindrops or snow, and rainSpeed ​​controls the falling speed of raindrops or snow; The severe weather effect of fog is simulated through the atmospheric scattering model, and the formula of the atmospheric scattering model is: I(x)=I0(x)·e -β·d(x) +A·(1-e -β·d(x) ) Among them, I0(x) is the pixel intensity when there is no fog, β is the atmospheric scattering coefficient, d(x) is the distance from the pixel to the camera, A is the atmospheric light intensity, and β and A are used to adjust the concentration and brightness of the fog.

3. The method for detecting targets in bad weather according to claim 1, characterized in that: The method of using CycleGan to input a real scene image and the combined image as training samples, and obtaining a second synthetic severe weather image paired with the clean image through training, comprises: When training the training samples through the CycleGan, the generator and the discriminator are used to perform domain conversion, and the training is evaluated through the loss function of the CycleGan; The parameters of the generator and the training are adjusted to control the strength and characteristics of the second synthetic severe weather image when it is generated.

4. The method for detecting targets in bad weather according to claim 3, characterized in that: The loss function of CycleGan includes adversarial loss, cycle consistency loss and identity mapping loss.

5. The method for detecting targets in bad weather according to claim 1, characterized in that: The CME-YOLO framework is used to perform upsampling processing on the training data set to obtain a sampling result, including: Inserting new data points into the original data points of the training data set using bilinear interpolation; Learnable convolution is used to find the best sampling area according to the data characteristics of the training data set, and sampling is performed according to the best sampling area to obtain the sampling result.

6. The method for detecting targets in bad weather according to claim 1, characterized in that: The sampling results are subjected to multimodal fusion and CPTFusion to obtain fusion features, including: The feature tensors of the visible light image and infrared image in the sampling results are reduced in dimension and reshaped through the CME-YOLO framework to obtain the embeddings; The embeddings are spliced ​​to obtain a unified tensor, and the residual bottleneck structure and bifurcation fusion module of the CME-YOLO framework are used to perform feature fusion and detail expression of visible light images and infrared images on the unified tensor. In addition, a Transformer variant module is used to perform feature enhancement and complex relationship capture processing on the feature fusion of visible light images and infrared images through a self-attention mechanism and a multi-layer perceptron to obtain fused features.

7. The method for detecting targets in bad weather according to claim 6, characterized in that: The method further comprises: The convolutional features of visible light images and infrared images are extracted through the network middle layer of the CME-YOLO framework, and the convolutional features are converted into sequences and rearranged to obtain visible light sequences and infrared sequences. The visible light sequence, infrared sequence and learnable positional encoding are combined to obtain the input sequence, where the input sequence passes through the difference bottleneck structure, the fork fusion module and the Transformer variant module in sequence; The input sequence is processed by reducing the dimension first and then increasing the dimension through two convolutional layers of different sizes through the residual bottleneck structure; The input sequence is grouped and convolved through the fork-fusion module to obtain local features and global features captured between groups. At the same time, the self-attention mechanism is used to strengthen the interaction between local features and global features captured between groups. The input sequence is transformed into query, key and value by three linear transformations of self-attention mechanism through Transformer variant module, attention weight is calculated according to query, key and value, and processed by linear transformation and activation function of multi-layer perceptron.

8. A device for detecting targets in bad weather, characterized in that: include: The first module is used to obtain a scene image of a target area, identify the scene image using a target detection model, and obtain a multispectral target detection result of the target area; The training process of the target detection model includes: The second module is used to obtain clean images and real scene severe weather images under different illuminations, synthesize the clean images to obtain a first synthesized severe weather image, and combine the real scene severe weather image and the first synthesized severe weather image to obtain a combined image; The third module is used to use CycleGan to input the real scene image and the combined image as training samples, and obtain a second synthetic severe weather image paired with the clean image through training; A fourth module is used to combine the second synthetic severe weather image and the real scene image into a training data set, wherein the training data set includes a visible light image and an infrared image; The fifth module is used to perform upsampling processing on the training data set using the CME-YOLO framework to obtain sampling results, perform multimodal fusion on the sampling results through CPTFusion to obtain fusion features, and detect the fusion features to obtain detection results, and complete the training of the target detection model according to the detection results.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the target detection method in severe weather as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the target detection method in severe weather as described in any one of claims 1-7.

Images