Environment detection method and system based on multi-modal data fusion and deep learning

Abstract

The invention provides an environment detection method and system based on a sample target detection model. The method comprises the following steps: synchronously acquiring an environment image, a video stream and physical parameters by using a multi-mode sensor; decomposing the data into image features and environmental parameter components through a dual-time sequence control signal, and realizing space-time alignment by adopting a linear phase filter; constructing a foreground region template based on the depth information, and generating target recognition feature representation containing an abnormal blurred target; adversarial training is carried out on the lightweight target detection network in combination with a transfer learning strategy, the network integrates convolutional features and a Transform attention mechanism, and the weight is dynamically adjusted through environmental parameters; fusing a target result and sensor data in real-time detection, and inputting a decision tree model for risk grading; and after the early warning is triggered, reconstructing a false detection sample through an online learning mechanism and iteratively optimizing the model. The system correspondingly comprises a multi-modal data acquisition module, a data enhancement and annotation module, a model training module, a real-time detection and fusion module and an early warning and optimization module. According to the invention, through multi-source data fusion, dynamic data enhancement and an adaptive compensation mechanism, the small target detection precision, the environmental adaptability and the real-time early warning capability are significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of environmental detection technology, and in particular to an environmental detection method and system based on multimodal data fusion and deep learning. Background Art

[0002] With the growing demand for environmental monitoring, traditional environmental monitoring methods suffer from low accuracy and poor adaptability to complex environments. A single sensor or detection method struggles to fully and accurately capture environmental information, and detection effectiveness declines significantly in harsh environmental conditions (such as fog and at night). Furthermore, existing target detection models are prone to missed and false detections when dealing with small and overlapping targets, failing to meet the requirements for real-time, accurate environmental monitoring. Therefore, there is an urgent need for an environmental monitoring method and system that can integrate multimodal data and adapt to complex environments. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide an environment detection method and system based on multimodal data fusion and deep learning to solve the above-mentioned technical problems.

[0004] The main purpose of this application is to provide an environment detection method based on a sample target detection model, comprising the following steps: Use multimodal sensors to synchronously collect environmental images, video streams and physical parameters to form collected data; Decomposing the collected data into an image feature component and an environmental parameter component by a data separation unit; alternately outputting the image feature component and the environmental parameter component at a first timing control signal frequency to generate a fusion signal; Performing standardized sampling and digital conversion on the environmental parameters in the fusion signal; alternately decoupling and outputting the digital representation of the image features and the digital representation of the environmental parameters at the frequency of the second timing control signal; A half-sampling period delay is applied to the digital representation of environmental parameters, and a linear phase filter is used to maintain spatiotemporal consistency. The clock time aligned in the time dimension is used as the output annotation benchmark for image features and environmental parameters. Generate a foreground area template based on the depth information of the 3D image data, extract the dynamic target as the target area, construct a logical target recognition feature representation in the target area, modify the local logical relationship of the digital representation of the image features, and generate a target recognition feature representation that includes abnormal blurred targets; The target detection network is trained adversarially using target recognition features, which represent abnormal blurred targets extracted from foreground region templates, and fine-tuned in stages using a transfer learning strategy. Inputting the real-time image into the trained target detection network to output the target detection result; screening the digital representation of the environmental parameters aligned in time sequence based on the foreground area template; fusing the digital representation of the physical parameters with the detection result and inputting it into the decision tree model; The decision tree model is combined with the feature weights output by the adaptive compensation module to perform risk classification. After the warning is triggered, the false positive samples are reconstructed into target recognition feature representations. The training set is added through the online learning mechanism to iteratively optimize the network weights.

[0005] Furthermore, when using target recognition features to conduct adversarial training on the target detection network, it includes: Construct an improved lightweight target detection network, fuse convolutional features with Transformer attention weighted features in the target detection network I; dynamically adjust the network weights II by parameter prediction of network input environment parameters.

[0006] Furthermore, the staged fine-tuning combined with the transfer learning strategy includes: a) Loading environment-specific datasets based on pre-trained models; b) Using cosine annealing learning rate adjustment strategy; c) Injecting adversarial samples generated by logically suspected targets; Improving detection robustness of small objects in super-resolution reconstruction through adversarial training.

[0007] Furthermore, the trained object detection network has a dynamic data enhancement strategy: Use OpenCV library to add simulated rain and fog effects; Use SRGAN super-resolution reconstruction algorithm to reconstruct small target areas; Logical suspected target injection: Modify image contours or segmentation information to generate suspected target samples.

[0008] Furthermore, the image preprocessing operation is also included before the separation unit decomposes the collected data: Use bilateral filtering algorithm to remove noise; Contrast-constrained adaptive histogram equalization is used for illumination equalization.

[0009] The present invention also provides an environment detection system based on a sample target detection model, comprising: Multimodal data acquisition module with: Input end, used for synchronously receiving image data, video stream and physical parameters; A data separation unit, used for decomposing a data stream into an image feature component and an environmental parameter component; A first dynamic switching unit, configured to alternately output fusion signals under a first timing control signal; Data enhancement and annotation module, with: A logical suspected target generation unit, configured to modify local logical relationships based on image feature representation to generate abnormal blurred targets; A timing compensation unit for achieving time alignment by applying a half-sampling period delay through a phase-holding filter; The model training module uses the target recognition feature representation to optimize the network's sensitivity to small targets through adversarial training units; The real-time detection and fusion module uses layered detection units and filters sensor data based on foreground area templates to detect 3D physical parameter anomalies in areas without 2D suspected targets. The early warning and optimization module uses a decision tree model combined with the feature weights output by the adaptive compensation unit to perform risk classification. After the early warning is triggered, the false positive samples are reconstructed into target recognition feature representations. The network weights are iteratively optimized by adding the training set through the online learning mechanism, and the feature representation reconstruction unit is used to convert the false positive samples into target recognition feature representations and add them to the training set.

[0010] Furthermore, the operating frequency of the first timing control signal is dynamically adjusted by a GPU-CUDA parallel computing framework.

[0011] Furthermore, the phase-preserving filter of the timing compensation unit exhibits a linear phase response, ensuring that the time dimension of the environmental parameters is aligned with the image features.

[0012] Furthermore, the adversarial training unit works in conjunction with the logical suspected target generation unit to improve the model's anti-interference ability by injecting abnormal virtual targets.

[0013] Furthermore, the adversarial training unit includes: Optimize the feature fusion unit to fuse convolutional features and Transformer attention weighted features in the neck network; The adaptive compensation unit dynamically adjusts the network weights by predicting the network input environment parameters.

[0014] The present invention has the following beneficial effects: improving detection accuracy: through multimodal data fusion and improved target detection model, the detection ability of small targets and overlapping targets is enhanced, and combined with dynamic data enhancement and adaptive compensation mechanism, the accuracy of the model in complex environments is improved.

[0015] Enhanced environmental adaptability: A data augmentation strategy that simulates harsh environments and an adaptive compensation module that dynamically adjusts model weights based on environmental parameters enable the system to adapt to different environmental conditions such as lighting and weather.

[0016] Real-time early warning and optimization: Real-time detection and multi-source data fusion can quickly determine the environmental risk level and issue timely warnings. The online learning mechanism realizes continuous optimization of the model and improves the detection performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a schematic diagram of the system framework principle in the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] Example 1: Reference Figure 1 First, multimodal sensors are deployed appropriately based on the detection environment to ensure comprehensive and accurate data collection. During the data collection phase, images, video streams, and physical parameters are synchronously collected at a set frequency, and images are pre-processed in a timely manner. During data enhancement and annotation, simulation environment parameters and small object processing methods are adjusted according to actual needs, and semi-automatic annotation tools are fully utilized to improve annotation efficiency.

[0020] During model training, appropriate training equipment and parameters are selected based on computing resources and data size, and model training is performed strictly according to a phased training and cosine annealing learning rate adjustment strategy. During real-time detection, GPU acceleration technology is effectively utilized to ensure real-time detection. When warning conditions occur during system operation, relevant samples are promptly processed and recorded for model optimization. Through continuous practice and optimization, the environmental detection method and system described in this invention achieve optimal performance.

[0021] Based on the characteristics of the environmental scene, the OpenCV library is used to add noise data simulating rain, fog, and low light to enhance the robustness of the model to harsh environments; a super-resolution reconstruction algorithm (such as SRGAN) is used to reconstruct small target areas and perform local magnification and annotation.

[0022] The main purpose of the present application is to provide an environment detection method based on a sample target detection model, comprising the following steps: using a multimodal sensor to synchronously collect environmental images, video streams and physical parameters to form collected data; decomposing the collected data into image feature components and environmental parameter components through a data separation unit; alternately outputting image feature components and environmental parameter components at a first timing control signal frequency to generate a fusion signal; performing standardized sampling and digital conversion on the environmental parameters in the fusion signal; alternately decoupling and outputting image feature digital representations and environmental parameter digital representations at a second timing control signal frequency; applying a half-sampling period delay to the environmental parameter digital representation, and using a linear phase filter to maintain spatiotemporal consistency; using a clock moment aligned in the time dimension as an output annotation benchmark for image features and environmental parameters; and generating a foreground area template based on the depth information of the three-dimensional image data. , extract dynamic targets as target areas, construct logical target recognition feature representations in the target areas, modify the local logical relationships of the digital representations of image features, and generate target recognition feature representations containing abnormal blurred targets; use target recognition features to conduct adversarial training on the target detection network, the target recognition feature representations contain abnormal blurred targets extracted from the foreground area templates, and are fine-tuned in stages in combination with the transfer learning strategy; input the real-time image into the trained target detection network to output the target detection results; based on the foreground area templates, screen the time-aligned digital representations of environmental parameters; fuse the digital representations of physical parameters with the detection results and input them into the decision tree model; the decision tree model performs risk classification in combination with the feature weights output by the adaptive compensation module; after triggering the warning, the false detection samples are reconstructed into target recognition feature representations; the network weights are iteratively optimized by adding the training set through the online learning mechanism.

[0023] In some embodiments, multimodal sensors (such as cameras, infrared sensors, and air quality sensors) are used to synchronously capture environmental images, video streams, and physical parameters (such as PM2.5 concentration, temperature and humidity, and light intensity) to form raw captured data. Synchronous acquisition ensures the temporal correlation between images and environmental parameters, laying the foundation for subsequent fusion analysis. Before data separation, the image data is preprocessed: a bilateral filtering algorithm is used for denoising. This algorithm simultaneously considers pixel spatial distance and grayscale differences, removing noise while retaining image edge information, avoiding the blurred target outline caused by traditional filtering. Contrast-limited adaptive histogram equalization (CLAHE) is used for illumination equalization, dividing the image into multiple sub-blocks for separate processing. By limiting contrast to prevent local over-enhancement, it effectively solves the problem of uneven illumination and improves the readability of the image under different lighting conditions.

[0024] In some embodiments, a data separation unit decomposes the collected data into image feature components and environmental parameter components, achieving a preliminary decoupling of visual information from physical parameters. At a first timing control signal frequency, the image feature components and environmental parameter components are alternately output to generate a fused signal. The operating frequency of this timing control signal is dynamically adjusted by the GPU-CUDA parallel computing framework, optimizing the data output rhythm in real time based on hardware computing power to ensure that data transmission efficiency matches subsequent processing capabilities.

[0025] In some embodiments, the environmental parameters in the fused signal are subjected to standardized sampling and digital conversion to convert continuous physical parameters into discrete digital signals for ease of computer processing.

[0026] In some embodiments, at the second timing control signal frequency, the digital representation of image features and the digital representation of environmental parameters are alternately decoupled and output to achieve further separation of the two types of data, preparing for subsequent timing alignment.

[0027] In some embodiments, a half-sampling period delay is applied to the digital representation of the environmental parameter, and a linear phase filter is used to maintain temporal and spatial consistency. The characteristics of the linear phase filter ensure that the phase delay of each frequency component of the signal is the same when it passes through, avoiding temporal and spatial misalignment caused by phase distortion.

[0028] In some embodiments, the clock moments aligned in the time dimension are used as the output annotation benchmarks for image features and environmental parameters to ensure that the two types of data are strictly aligned on the time axis, providing a unified time reference for subsequent fusion analysis.

[0029] In some embodiments, the foreground area template is constructed based on the depth information of the three-dimensional image data to generate the foreground area template, and the dynamic target is extracted as the target area through depth threshold segmentation to eliminate background interference.

[0030] In some embodiments, logical feature reconstruction involves constructing a logical target recognition feature representation within the target region. By modifying the local logical relationships within the digital representation of image features, a target recognition feature representation containing abnormally blurred targets is generated. This process simulates interfering targets (such as reflections and shadows) that may appear in a real environment, providing material for adversarial training.

[0031] In some embodiments, the adversarial training strategy uses target recognition features containing abnormally blurred targets to perform adversarial training on the target detection network, and enhances the robustness of the model to complex scenarios by introducing adversarial samples.

[0032] In some embodiments, staged fine-tuning of transfer learning involves loading a specific environment dataset on top of a pre-trained model (e.g., YOLOv8 pre-trained on the COCO dataset), leveraging the pre-trained model's generalization capabilities to accelerate convergence. A cosine annealing learning rate adjustment strategy dynamically adjusts the learning rate, rapidly decreasing it early in training to escape local optima and then slowly oscillating it later to fine-tune parameters. Adversarial examples generated by logically suspected targets are injected to enhance detection capabilities for small objects in super-resolution reconstruction through adversarial training.

[0033] In some embodiments, real-time detection and data fusion include: Target detection: Inputting real-time images into a trained target detection network, which outputs the target category, location, and confidence level. The GPU-CUDA parallel computing framework is used to accelerate the forward propagation process, ensuring real-time detection. Data screening and fusion: Using foreground region templates, digital representations of environmental parameters are screened for temporal alignment. The digital representations of physical parameters are then fused with the detection results and input into a decision tree model. The decision tree model combines the feature weights output by the adaptive compensation module to perform risk classification. These weights dynamically adjust the importance of each feature based on environmental parameters (such as lighting and fog density), improving the accuracy of risk assessment.

[0034] In some embodiments, risk warning includes triggering a warning when a high-risk target is detected, pushing it to the management platform through sound and light alarms and the network, and displaying the warning location, target category and risk level in real time.

[0035] In some embodiments, the online learning mechanism involves reconstructing false positives into target recognition feature representations, adding a training set to the online learning mechanism, and iteratively optimizing network weights. This mechanism enables the model to continuously learn from real-world applications, continuously improving detection performance and forming a closed loop of "detection, feedback, and optimization."

[0036] In the above-mentioned embodiments, the temporal alignment and feature fusion of images and physical parameters overcomes the limitations of a single sensor. For example, in foggy scenes, combining infrared temperature parameters with image features can more accurately identify heat source targets, avoiding missed detections due to low visibility. By injecting unusually blurred targets and simulating harsh environments (such as adding rain and fog effects using OpenCV), the model's ability to resist interference factors is enhanced. The combination of a cosine annealing learning rate and online learning ensures training efficiency while enabling continuous model evolution. This allows the model to adapt to environmental changes over the long term. For example, when pollutant types change in an industrial park, the model can quickly adjust its detection strategy by learning from false positives.

[0037] In some embodiments, when using target recognition features to perform adversarial training on a target detection network, it includes: constructing an improved lightweight target detection network, fusing convolutional features and Transformer attention weighted features I in the target detection network; and dynamically adjusting network weights II by parameter prediction of network input environment parameters.

[0038] In some embodiments, during adversarial training of target detection networks, traditional models struggle to balance feature extraction efficiency with adaptability to complex environments. This is especially true when dealing with small and overlapping targets, where insufficient feature representation leads to decreased detection accuracy. This application aims to improve the model's ability to represent targets in complex environments, enhancing interference resistance and detection robustness by fusing convolutional features with the Transformer attention mechanism and dynamically adjusting network weights based on environmental parameters.

[0039] In some embodiments, a Transformer multi-head self-attention mechanism is introduced into the object detection network, fusing convolutional features with Transformer attention-weighted features in the core network (such as the feature fusion layer of YOLOv8). Convolutional neural networks (CNNs) excel at extracting local spatial features, while the Transformer self-attention mechanism captures global contextual relationships. Combining these two enhances the model's feature extraction capabilities for small objects (such as distant smoke particles) and overlapping objects (such as dense industrial pipes). In specific implementations, the feature map extracted by the CNN is input into the Transformer module, which calculates attention weights between pixels to highlight key target areas and suppress background noise. For example, in a hazy environment, the attention mechanism can focus on heat sources while ignoring interfering pixels caused by the haze. The number of model parameters can be reduced by optimizing the network structure (such as using depthwise separable convolutions and pruning redundant connections).

[0040] In some embodiments, parameter prediction network construction involves designing a parameter prediction network that inputs environmental parameters (such as light intensity, fog concentration, temperature and humidity) and outputs parameters used to adjust network weights. The network uses a fully connected layer structure and learns the mapping relationship between environmental parameters and optimal weights through training.

[0041] In some embodiments, the dynamic weight adjustment mechanism adjusts the weights of the convolutional and fully connected layers of the object detection network in real time based on the output of the parameter prediction network. For example, when a foggy environment is detected, the network automatically increases the weight of low-frequency features to suppress high-frequency noise caused by the fog; in nighttime scenes, the network increases the weight of the infrared feature channel to optimize target recognition in low-light conditions.

[0042] In some embodiments, adaptive compensation is the core of dynamic weight adjustment. By monitoring environmental parameter changes in real time, the update of the parameter prediction network is triggered, and the adjusted weights are applied to the detection model to achieve a closed-loop response of "environmental change-parameter perception-model adaptation".

[0043] In some embodiments, the staged fine-tuning combined with the transfer learning strategy includes: a) loading an environment-specific dataset based on the pre-trained model; b) adopting a cosine annealing learning rate adjustment strategy; c) injecting adversarial samples generated by logically suspected targets; and enhancing the detection robustness of small targets reconstructed in super-resolution through adversarial training.

[0044] In some embodiments, pre-trained model loading and environmental dataset adaptation include pre-trained model selection, environment-specific dataset construction, and feature transfer mechanism construction. Specifically, pre-trained model selection is based on a target detection model (such as YOLOv8) pre-trained on a large-scale general dataset (such as COCO), which has learned general target feature representation capabilities (such as edges, textures, shapes, etc.). Environment-specific dataset construction is to collect datasets for specific environmental scenes (such as industrial parks, forests, city streets, etc.), which contain target images and corresponding environmental parameters (such as pollutant concentrations, lighting conditions, etc.). Dataset annotation is combined with semi-automatic annotation tools, using sensor data to assist in locating the target area and improve annotation efficiency. The feature transfer mechanism is to freeze the backbone network parameters of the pre-trained model and only fine-tune the classification layer and regression layer parameters, so that the model can learn environment-specific target representations while retaining general feature extraction capabilities.

[0045] In some embodiments, adversarial sample injection and small target detection enhancement include: Logical suspected target generation: suspected target samples are generated by modifying image contours or segmentation information to simulate interference factors in the actual environment (such as false targets formed by reflections and shadows) as adversarial sample injection training process. Super-resolution reconstruction of small target processing: The SRGAN algorithm is used to perform super-resolution reconstruction of small target areas in the dataset to enhance the detailed features of small targets (such as fine smoke particles and cracks in distant pipes), and the reconstructed samples are added to the training set to improve the model's detection ability for small targets. Adversarial training process: During the training process, real samples and adversarial samples are input alternately to force the model to learn to distinguish between real targets and interference factors and enhance robustness. For example, during training, normal images and images with simulated rain and fog noise are input at the same time to train the model's detection capabilities under harsh conditions.

[0046] In some embodiments, the trained target detection network has a dynamic data enhancement strategy: using the OpenCV library to add simulated rain and fog effects; using the SRGAN super-resolution reconstruction algorithm to reconstruct small target areas; logical suspected target injection: modifying image contours or segmentation information to generate suspected target samples.

[0047] In some embodiments, rain and fog effects are simulated by adding simulated rain and fog noise using image processing functions from the OpenCV library. Specifically, these include: Rain line simulation: Generating random straight line segments and overlaying them onto the image to simulate rainfall scenarios of varying intensities. The density, length, and transparency of the line segments can be adjusted based on the rain intensity parameter. Fog effects: Combining Gaussian blur with grayscale transformation reduces image contrast and simulates fog of varying concentrations. The degree of fog is controlled by adjusting the blur kernel size and gamma coefficient. Low-light enhancement: By reducing the overall image brightness and adding localized shadows, low-light environments such as nighttime or tunnels are simulated, enhancing the model's ability to recognize targets in low-light conditions.

[0048] In some embodiments, the SRGAN algorithm for super-resolution reconstruction of small object areas uses a generative adversarial network (GAN) architecture, consisting of a generator and a discriminator. The generator is responsible for reconstructing low-resolution small object images into high-resolution images, enhancing details through deconvolution and residual blocks. The discriminator distinguishes the reconstructed image from the true high-resolution image, guiding the generator to produce details that are closer to reality. The small object processing process includes: screening small object images from the dataset where the target pixel ratio is less than a threshold (such as 5%); using SRGAN to super-reconstruct the small object area, increasing the resolution by 2-4 times; and adding the reconstructed image to the training set while retaining the original small object image to form multi-scale training samples.

[0049] In some embodiments, the suspected target generation method includes: contour modification: using image editing algorithms (such as OpenCV's contour operations) to modify the contour shape of a real target to generate a similar but unreal target (such as distorting the outline of a circular pipe into an irregular shape); segmentation information tampering: modifying the semantic segmentation mask of the image to add false target segmentation areas to the background area to simulate false targets caused by sensor noise or image processing errors. Injection strategy: mixing logical suspected target samples into normal training data at a certain ratio (such as 10%-20% of the training set) to force the model to learn to distinguish between real targets and logically fabricated interference targets, thereby improving anti-interference capabilities.

[0050] In some embodiments, before the separation unit decomposes the collected data, an image preprocessing operation is further included: using a bilateral filtering algorithm for denoising; and using contrast-limited adaptive histogram equalization for illumination equalization.

[0051] In some embodiments, the bilateral filtering algorithm adopts a traditional nonlinear filtering algorithm while considering both the spatial distance and grayscale difference of pixels.

[0052] In some embodiments, contrast-limited adaptive histogram equalization includes the following steps: dividing the image into multiple non-overlapping sub-blocks (e.g., 8×8 pixels) and performing histogram equalization on each sub-block; preventing excessive amplification of noise within the sub-block due to equalization by limiting the contrast (usually setting a threshold of 40); and using bilinear interpolation to process sub-block boundaries to avoid blocking effects and achieve global illumination equalization.

[0053] Example 2: Reference Figure 2 The present invention also provides an environmental detection system based on a sample target detection model, comprising: a multimodal data acquisition module having: an input end for synchronously receiving image data, video stream and physical parameters; a data separation unit for decomposing the data stream into image feature components and environmental parameter components; a first dynamic switching unit for alternately outputting fusion signals under a first timing control signal.

[0054] The data enhancement and annotation module has: a logical suspected target generation unit, which is used to modify the local logical relationship based on the image feature representation to generate abnormal blurred targets; a timing compensation unit, which is used to achieve time alignment by applying a half-sampling period delay through a phase-holding filter.

[0055] The model training module optimizes the network's sensitivity to small targets by using the target recognition feature representation through the adversarial training unit.

[0056] The real-time detection and fusion module uses layered detection units and filters sensor data based on foreground area templates to perform 3D physical parameter anomaly detection in areas without 2D suspected targets.

[0057] The early warning and optimization module uses a decision tree model combined with the feature weights output by the adaptive compensation unit to perform risk classification. After the early warning is triggered, the false positive samples are reconstructed into target recognition feature representations. The network weights are iteratively optimized by adding the training set through the online learning mechanism, and the feature representation reconstruction unit is used to convert the false positive samples into target recognition feature representations and add them to the training set.

[0058] In some embodiments, the input end is used to synchronously receive image data, video streams and physical parameters, supports the access of multiple sensors (such as RGB cameras, infrared cameras, temperature and humidity sensors, gas sensors, etc.), and has data buffering and synchronous triggering mechanisms to ensure the consistency of timestamps of different types of data.

[0059] The data separation unit uses signal processing algorithms to decompose the input data stream into image feature components and environmental parameter components, and effectively separates the two types of data through frequency domain analysis or feature space projection, laying the foundation for subsequent time series processing.

[0060] The first dynamic switching unit alternately outputs fused signals under the first timing control signal. Controlled by the GPU-CUDA parallel computing framework, this unit dynamically adjusts the switching frequency based on current computing resources to avoid data congestion or processing delays. For example, if the GPU load is detected to be too high, the timing frequency is automatically reduced to ensure priority processing of critical data.

[0061] The logical suspected target generation unit modifies the local logical relationship based on the image feature representation to generate abnormal blurred targets. It simulates interference factors in the actual environment (such as reflective objects and shadow areas) by means of contour deformation and segmentation area tampering, providing diverse negative samples for adversarial training.

[0062] The timing compensation unit achieves time alignment by applying a half-sampling period delay through a phase-preserving filter. The filter exhibits a linear phase response, ensuring that the environmental parameters and image features are strictly aligned in the time dimension.

[0063] The adversarial training unit utilizes target recognition feature representations to optimize the network's sensitivity to small targets. By inputting features containing unusually blurred targets into the network, it forces the model to learn to distinguish between real targets and interference factors. This unit works in conjunction with the logically suspected target generation unit, forming a closed "generation-training-feedback" loop to continuously improve the model's anti-interference capabilities.

[0064] The layered detection unit filters sensor data based on foreground area templates and performs 3D physical parameter anomaly detection in areas without 2D suspected targets. This mechanism implements dual verification through "visual detection + physical parameter analysis." For example, in a forest monitoring scenario, if no open fire is detected in the image but the infrared temperature parameters are abnormal, further thermal imaging analysis is triggered to avoid missed detections due to foliage obstruction.

[0065] The decision tree model combines the feature weights output by the adaptive compensation unit to perform risk classification. Trained with historical data, the model comprehensively determines risk levels based on target category, confidence level, and environmental parameters. For example, by combining excessive chemical gas concentrations with image targets in the corresponding area, the model can determine whether there is a leak risk and classify it accordingly.

[0066] After triggering an alert, the online learning mechanism reconstructs false positive samples into target recognition feature representations. The feature representation reconstruction unit then converts these false positive samples into target recognition feature representations and adds them to the training set, enabling iterative optimization of network weights. This mechanism enables the system to self-evolve. For example, in industrial inspection, as new defect types emerge, the model can quickly update detection rules by learning from false positive samples.

[0067] In some embodiments, the operating frequency of the first timing control signal is dynamically adjusted by the GPU-CUDA parallel computing framework. In the process of multimodal data fusion, the alternating output frequency of the image feature component and the environmental parameter component directly affects the efficiency and real-time performance of data fusion. Traditional fixed-frequency timing control cannot adapt to changes in different hardware computing power and data flow, which may cause data buffer overflow or idle processing resources. The GPU-CUDA dynamic adjustment mechanism proposed in this claim optimizes the operating frequency of the first timing control signal in real time by utilizing the parallel computing capability of the GPU, thereby ensuring the efficient operation of the data processing process.

[0068] In some embodiments, the GPU-CUDA parallel computing framework utilizes NVIDIA's CUDA platform to distribute the frequency calculation tasks of timing control signals to multiple stream processors of the GPU for parallel processing. The CUDA core's high-speed floating-point computing capabilities enable real-time monitoring of hardware indicators such as the current GPU load and memory bandwidth. The dynamic frequency adjustment strategy establishes a mapping model between hardware resources and timing frequencies. For example, when GPU utilization is below 30%, the timing frequency is increased to fully utilize computing power; when utilization exceeds 80%, the frequency is reduced to avoid overload. A feedback control algorithm is used to dynamically adjust the frequency based on the fill status of the data buffer, maintaining the buffer at a reasonable level (e.g., 50%-70%) to avoid data loss or processing delays.

[0069] The implementation process includes: initialization phase: preset the benchmark timing frequency according to the GPU model; operation phase: real-time collection of GPU core temperature, video memory occupancy, number of parallel threads and other parameters; decision phase: calculate the optimal frequency through preset threshold rules or machine learning models; adjustment phase: update the control signal frequency of the first dynamic switching unit through CUDAAPI.

[0070] In some embodiments, the phase-preserving filter of the timing compensation unit exhibits a linear phase response to ensure that the time dimension of the environmental parameter is aligned with the image feature. A half-sampling period delay is applied to the digital representation of the environmental parameter to compensate for the sampling frequency difference between it and the image feature.

[0071] A linear phase filter, whose impulse response exhibits even or odd symmetry, ensures linear phase response. Passing the environmental parameter signal through this filter eliminates phase offsets caused by delays and transmission, achieving time alignment with image features. Using a unified clock source as the time reference, precise timestamps are added to image features and environmental parameters. After filter compensation, the output is labeled using the time-aligned clock moment, ensuring a one-to-one correspondence between the two data types on the time axis.

[0072] In some embodiments, the adversarial training unit works in conjunction with a logical suspected target generation unit to improve the model's anti-interference ability by injecting abnormal virtual targets.

[0073] In some embodiments, the interference sample generation and training iteration process includes: the logical suspected target generation unit generates targeted abnormal blurred targets (such as the shape of reflective areas that the model is prone to misdetection) based on the weakness analysis of the current model; the adversarial training unit injects these samples into the training process, forcing the model to learn to distinguish between real targets and interference; during the training process, the model's response to various interference samples is recorded and fed back to the logical suspected target generation unit; the generation unit adjusts the sample generation strategy based on the feedback, such as increasing the interference types with high false detection rates of the model, to form an iterative optimization closed loop.

[0074] In some embodiments, the adversarial training unit locates the model's sensitive areas to specific interference patterns through gradient analysis or activation value visualization; the logical suspected target generation unit modifies the image contour or segmentation information based on these weaknesses to generate more challenging interference samples; for example, if the model has a high false detection rate for arc-shaped reflective areas, the generation unit focuses on generating arc-shaped interference samples with different curvatures and brightness.

[0075] By continuously injecting diverse interference samples, the model's feature extraction layer gradually learns to suppress interference features and enhance true target features. The collaborative mechanism enables the model to adapt to various interference factors that may appear in the actual environment during the training phase, reducing the false detection rate after deployment.

[0076] Furthermore, the adversarial training unit includes: an optimized feature fusion unit that fuses convolutional features with Transformer attention weighted features in the neck network; in the neck network of the object detection network, the convolutional features extracted by CNN are fused with the multi-head self-attention features of Transformer; the convolutional features retain the local details of the target (such as texture and edges), and the Transformer attention features capture the global contextual relationship (such as the semantic relationship between the target and the background). The fusion of the two is achieved through splicing or weighted summation; The adaptive compensation unit dynamically adjusts the network weights by predicting the network input environment parameters.

[0077] Construct a fully connected neural network, input environmental parameters (such as light intensity, fog concentration, temperature and humidity, etc.), and output parameters used to adjust network weights; the network structure is designed according to the dimensions of environmental parameters. For example, input 3D environmental parameters (temperature, humidity, light) and output 5D weight adjustment parameters.

[0078] Dynamic weight adjustment mechanism: Based on the output of the prediction network, the weights of the convolutional and fully connected layers of the target detection network are adjusted in real time. For example, when a foggy environment is detected, the weight of the low-frequency feature channel is increased to reduce the impact of high-frequency noise caused by the fog. In night scenes, the weight corresponding to the infrared feature is increased to optimize low-light detection effects.

[0079] In some embodiments, the present application improves the integrity of environmental information through multimodal data fusion and timing alignment technology. Specifically, multimodal sensors synchronously collect images, video streams, and physical parameters (such as temperature, humidity, and PM2.5 concentration). The data separation unit decomposes the data into image features and environmental parameter components, and achieves fusion and decoupling through dual timing control signals. This technical feature improves detection effectiveness through the following mechanisms: Temporal and spatial consistency assurance: A half-sampling period delay is applied to the environmental parameters and a linear phase filter is used to ensure that physical parameters such as temperature and gas concentration are strictly aligned with the image frame in the time dimension.

[0080] Multi-dimensional information complementation: Image features provide visual representations of targets (such as shape and texture), while environmental parameters complement physical properties (such as temperature anomalies and gas leaks). The fusion of these two overcomes the limitations of single sensors. For example, in foggy scenes, combining infrared temperature parameters with visible light images can accurately identify heat sources hidden in the fog.

[0081] Optimized data fusion efficiency: The frequency of the primary timing control signal is dynamically adjusted by GPU-CUDA, optimizing the data output rhythm in real time based on the hardware's computing power. When the detection equipment is overloaded, the data switching frequency is automatically reduced to ensure that critical environmental parameters are prioritized and guarantee system stability in high-concurrency scenarios.

[0082] In some embodiments, the improved lightweight target detection network of the present application enhances the detection capabilities of small targets and overlapping targets, constructs a lightweight network that integrates convolutional features and Transformer attention mechanism, and improves detection performance through the following technical innovations: Feature representation capability upgrade: Integrate CNN local feature extraction capabilities and Transformer global attention mechanism in the neck network.

[0083] Dynamic weight adaptive adjustment: The parameter prediction network uses environmental inputs (such as light intensity and fog density) to adjust network weights in real time. In nighttime scenes, the infrared feature channel weight is automatically increased to enhance target recognition in low light conditions. In foggy environments, the weight of low-frequency features is increased to suppress high-frequency noise caused by fog.

[0084] In some embodiments, the present application improves the model environment robustness through a dynamic data enhancement strategy. Specifically, diverse training samples are constructed through OpenCV simulation of harsh environments, SRGAN super-resolution reconstruction, and logical suspected target injection.

[0085] Simulating harsh environments: When using OpenCV to add rain and fog noise, random straight line segments are used to simulate changes in rainfall intensity. Gaussian blurring combined with grayscale transformation generates fog of varying concentrations, allowing the model to adapt to extreme weather conditions during training.

[0086] Small object detail enhancement: SRGAN performs super-resolution reconstruction on small object areas, increasing the resolution by 2-4 times before adding them to the training set.

[0087] Enhanced anti-interference capabilities: By modifying image contours or segmentation information, we generate logically plausible targets (such as false targets caused by reflections) and mix them into the training data at a ratio of 10%-20%. Through adversarial training, the model learns to distinguish between real targets and interference factors.

[0088] In the foregoing specification, examples have been described with reference to specific exemplary embodiments. However, it will be apparent that various modifications and changes can be made to the specific examples without departing from the scope as set forth in the appended claims, and the claims are not limited to the specific examples described above.

Claims

1. The environment detection method based on the sample target detection model is characterized by: The steps include: Use multimodal sensors to synchronously collect environmental images, video streams and physical parameters to form collected data; Decomposing the collected data into an image feature component and an environmental parameter component by a data separation unit; alternately outputting the image feature component and the environmental parameter component at a first timing control signal frequency to generate a fusion signal; Performing standardized sampling and digital conversion on the environmental parameters in the fusion signal; alternately decoupling and outputting the digital representation of the image features and the digital representation of the environmental parameters at the frequency of the second timing control signal; A half-sampling period delay is applied to the digital representation of environmental parameters, and a linear phase filter is used to maintain spatiotemporal consistency. The clock time aligned in the time dimension is used as the output annotation benchmark for image features and environmental parameters. Generate a foreground area template based on the depth information of the 3D image data, extract the dynamic target as the target area, construct a logical target recognition feature representation in the target area, modify the local logical relationship of the digital representation of the image features, and generate a target recognition feature representation that includes abnormal blurred targets; The target detection network is trained adversarially using target recognition features, which represent abnormal blurred targets extracted from foreground region templates, and fine-tuned in stages using a transfer learning strategy. Inputting the real-time image into the trained target detection network to output the target detection result; filtering the digital representation of the time-series aligned environmental parameters based on the foreground area template; Fuse the digital representation of physical parameters with the test results and input them into the decision tree model; The decision tree model is combined with the feature weights output by the adaptive compensation module to perform risk classification; After the warning is triggered, the false detection samples are reconstructed into target recognition feature representation; The network weights are iteratively optimized by adding training sets through the online learning mechanism.

2. The environment detection method based on the sample target detection model according to claim 1, characterized in that: When using target recognition features to conduct adversarial training on the target detection network, it includes: Construct an improved lightweight target detection network, fuse convolutional features with Transformer attention weighted features in the target detection network I; dynamically adjust the network weights II by parameter prediction of network input environment parameters.

3. The environment detection method based on the sample target detection model according to claim 1, characterized in that: When combined with the transfer learning strategy for staged fine-tuning, it includes: a) Loading environment-specific datasets based on pre-trained models; b) Using cosine annealing learning rate adjustment strategy; c) Injecting adversarial samples generated by logically suspected targets; Improving detection robustness of small objects in super-resolution reconstruction through adversarial training.

4. The environment detection method based on the sample target detection model according to claim 1, characterized in that: The trained object detection network has a dynamic data enhancement strategy: Use OpenCV library to add simulated rain and fog effects; Use SRGAN super-resolution reconstruction algorithm to reconstruct small target areas; Logical suspected target injection: Modify image contours or segmentation information to generate suspected target samples.

5. The environment detection method based on the sample target detection model according to claim 1, characterized in that: Before the separation unit decomposes the acquired data, it also includes image preprocessing operations: Use bilateral filtering algorithm to remove noise; Contrast-constrained adaptive histogram equalization is used for illumination equalization.

6. An environment detection system based on a sample target detection model, characterized in that: include: Multimodal data acquisition module with: Input end, used for synchronously receiving image data, video stream and physical parameters; A data separation unit, used for decomposing a data stream into an image feature component and an environmental parameter component; A first dynamic switching unit, configured to alternately output fusion signals under a first timing control signal; Data enhancement and annotation module, with: A logical suspected target generation unit, configured to modify local logical relationships based on image feature representation to generate abnormal blurred targets; A timing compensation unit for achieving time alignment by applying a half-sampling period delay through a phase-holding filter; The model training module uses the target recognition feature representation to optimize the network's sensitivity to small targets through adversarial training units; The real-time detection and fusion module uses layered detection units and filters sensor data based on foreground area templates to detect 3D physical parameter anomalies in areas without 2D suspected targets. The early warning and optimization module uses a decision tree model combined with the feature weights output by the adaptive compensation unit to perform risk classification; After the warning is triggered, the false detection samples are reconstructed into target recognition feature representation; The network weights are iteratively optimized by adding the training set through the online learning mechanism, and the feature representation reconstruction unit is used to convert the false positive samples into target recognition feature representation and add them to the training set.

7. The environment detection system based on the sample target detection model according to claim 6, characterized in that: The operating frequency of the first timing control signal is dynamically adjusted by a GPU-CUDA parallel computing framework.

8. The environment detection system based on the sample target detection model according to claim 6, characterized in that: The phase-preserving filter of the timing compensation unit exhibits a linear phase response, ensuring that the environmental parameters are aligned with the time dimension of the image features.

9. The environment detection system based on the sample target detection model according to claim 6, characterized in that: The adversarial training unit works in conjunction with the logical suspected target generation unit to improve the model's anti-interference ability by injecting abnormal virtual targets.

10. The environment detection system based on the sample target detection model according to claim 6, characterized in that: The adversarial training unit includes: Optimize the feature fusion unit to fuse convolutional features and Transformer attention weighted features in the neck network; The adaptive compensation unit dynamically adjusts the network weights by predicting the network input environment parameters.

Images