A method, system, device and medium for pedestrian detection based on adverse weather

Abstract

The application provides a pedestrian detection method and system based on severe weather, equipment and medium, belonging to the technical field of data processing. The method comprises: acquiring a modulated light field sequence and a corresponding light intensity signal sequence; the modulated light field sequence is generated according to a plurality of sequentially arranged speckle patterns, the modulated light field sequence comprises a plurality of modulation patterns, and the light intensity signal sequence is time-synchronized with the modulated light field sequence; inputting the modulation patterns of the modulated light field sequence and the light intensity signal sequence into a target detection model, performing feature encoding and correlation calculation on the two sequences through a backbone network of the target detection model to obtain a correlation feature map representing light field correlation information; and obtaining a pedestrian detection result in the target area based on the target detection model and the correlation feature map. The application improves the efficiency and accuracy of pedestrian detection in severe weather.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a pedestrian detection method, system, device, and medium based on severe weather conditions. Background Technology

[0002] Pedestrian detection is an application of object detection algorithms. It not only provides core technical support for the autonomous driving services of new energy vehicles but also offers fundamental technical support for scenarios such as intelligent robot navigation, human-computer interaction, aerial image analysis, and motion behavior recognition. Currently, deep learning-based object detection algorithms, such as the two-stage Faster R-CNN and the single-stage YOLO series, have become mainstream solutions in this field. These methods automatically learn image features through convolutional neural networks and achieve excellent performance in clear scenes. To further improve efficiency and accuracy, industry research continues to evolve towards lightweight network structures, multi-scale feature fusion, and the introduction of attention mechanisms. For example, Faster R-CNN and YOLO are two different convolutional neural network algorithms for object detection. Convolutional neural network-based algorithms rely on post-processing with Non-Maximum Suppression (NMS). RT-DETR (Real-Time-Detection Transformer) is a novel end-to-end object detection framework that combines the Transformer architecture, achieving a good balance between speed and accuracy.

[0003] Faster R-CNN and YOLO algorithms significantly improved detection performance and evolved from a two-stage model to a one-stage model, but NMS post-processing still consumes considerable time. RT-DETR addresses the post-processing dependency of convolutional neural networks, achieving end-to-end detection—inputting images into the model, processing them, and outputting the final result. Furthermore, various optimizations were made based on Faster R-CNN, YOLO, and RT-DETR. These optimizations primarily focused on improving object detection accuracy at different scales and lightweighting the network.

[0004] However, the aforementioned technologies still face fundamental challenges when dealing with complex real-world environments, especially severe weather conditions such as fog, haze, rain, and snow. In these scenarios, suspended particles in the atmosphere cause severe light scattering and absorption, resulting in decreased contrast, blurred details, and increased noise in the acquired images. The visual features relied upon by traditional algorithms are significantly weakened or even obscured. This directly leads to a decline in the model's feature extraction capabilities, particularly for distant, occluded, or small-scale pedestrian targets, resulting in a significant increase in both false negative and false positive rates. Although new frameworks such as RT-DETR have improved post-processing efficiency, their detection performance is still inherently limited by the quality of the input image. Under extremely low visibility conditions, their accuracy and robustness are difficult to guarantee. Therefore, on the one hand, visual algorithms based on conventional imaging experience a sharp decline in performance under severe weather conditions; on the other hand, image enhancement or dehazing methods specifically designed for severe weather are often computationally complex, have poor real-time performance, and are difficult to integrate efficiently with high-speed target detection workflows. Summary of the Invention

[0005] In view of this, it is necessary to provide a pedestrian detection method, system, device and medium based on severe weather to solve the technical problems in the existing technology that feature extraction is difficult and detection accuracy is greatly reduced due to severe image quality deterioration in severe weather such as fog and haze.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a pedestrian detection method based on severe weather, comprising: Acquire a modulated light field sequence and a corresponding light intensity signal sequence; the modulated light field sequence is generated based on multiple sequentially arranged speckle patterns, the modulated light field sequence includes multiple modulation patterns, and the light intensity signal sequence is time-synchronized with the modulated light field sequence; The modulation pattern and the light intensity signal sequence of the modulation light field sequence are input into the target detection model. The backbone network of the target detection model performs feature encoding and correlation calculation on the two sequences to obtain a correlation feature map that represents the correlation information of the light field. Based on the target detection model, pedestrian detection results within the target area are obtained according to the associated feature map.

[0007] In one possible implementation, acquiring the modulated light field sequence and the corresponding light intensity signal sequence includes: A light beam is generated and emitted by a light source to a spatial light modulator; The spatial light modulator is controlled to generate the speckle pattern, and the received light beam is spatially modulated according to the speckle pattern to generate the corresponding modulation pattern in sequence, thereby obtaining the modulation light field sequence and projecting it onto the target area. The light intensity signal sequence is obtained by collecting the light signal reflected after passing through the target area using a photodetector.

[0008] In one possible implementation, the backbone network includes a first encoding network and a second encoding network. The process of using the backbone network of the target detection model to perform feature encoding and association calculations on the two sequences to obtain an association feature map representing the light field association information includes: Based on the first coding network, feature encoding is performed on each modulation pattern in the modulation light field sequence to obtain a first feature sequence; Based on the second coding network, feature encoding is performed on each light intensity value in the light intensity signal sequence to obtain a second feature sequence; The associated feature map is calculated based on the first feature sequence and the second feature sequence.

[0009] In one possible implementation, the backbone network further includes a convolutional neural network module, and the object detection model further includes a hybrid encoder connected to the output of the convolutional neural network module, the hybrid encoder including an adaptive interleaved feature interaction module and a cross-scale channel fusion module; the method further includes: Based on the convolutional neural network module, feature encoding and multi-scale downsampling are performed on the associated feature map to output multiple initial feature maps of different scales; Based on the adaptive interleaved feature interaction module, the initial feature map with the smallest scale is serialized to obtain the target feature; The target features are fused with initial feature maps of other scales based on the cross-scale channel fusion module to generate multi-scale fused features.

[0010] In one possible implementation, the target detection model further includes a query selector, a decoder, and a detection head; the method further includes: The multi-scale fusion features are interacted with a set of preset initial query vectors through attention to obtain a preliminary prediction score corresponding to each initial query vector; The query selector calculates the uncertainty score of each initial query vector based on the preliminary prediction score, and selects multiple candidate query vectors with the lowest uncertainty scores from the initial query vectors. The candidate query vector is decoded and refined based on the decoder to obtain the target query vector; Based on the detection head and the target query vector, the probability distribution of the pedestrian's category and the coordinates of the predicted bounding box representing the pedestrian's location are output, thereby generating the pedestrian detection result.

[0011] One possible implementation also includes: Construct a training dataset, which contains multiple sets of sample data collected under simulated or real severe weather conditions. Each set of sample data includes a modulated light field sequence, a time-synchronized integrated light intensity signal sequence, and corresponding labels labeled with pedestrian target categories and locations. With the goal of minimizing the prediction error, the initial model is iteratively trained end-to-end using the training dataset to obtain the object detection model; wherein, the prediction error is calculated based on the classification loss and the bounding box regression loss.

[0012] In one possible implementation, the speckle pattern is generated based on a Hadamard matrix.

[0013] Secondly, the present invention also provides a pedestrian detection system based on severe weather conditions, comprising: An acquisition module is used to acquire a modulated light field sequence and a corresponding light intensity signal sequence; the modulated light field sequence is generated based on multiple sequentially arranged speckle patterns, the modulated light field sequence includes multiple modulation patterns, and the light intensity signal sequence is time-synchronized with the modulated light field sequence; The encoding and calculation module is used to input the modulation pattern of the modulation light field sequence and the light intensity signal sequence into the target detection model, and to perform feature encoding and correlation calculation on the two sequences through the backbone network of the target detection model to obtain a correlation feature map that represents the correlation information of the light field. The detection and recognition module is used to obtain pedestrian detection results within the target area based on the target detection model and the associated feature map.

[0014] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the pedestrian detection method based on severe weather described in any of the above implementations.

[0015] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the pedestrian detection method based on severe weather described in any of the above implementations.

[0016] The beneficial effects of this invention are as follows: The pedestrian detection method based on severe weather provided by this invention firstly constructs a complementary dual-modal data stream based on synchronously acquired light intensity signal sequences (which can be regarded as changes in overall reflection intensity or average brightness) and modulation patterns (containing fine spatial structure modulation information). Light intensity signals are sensitive to rapid changes and large targets, while the modulated light field encodes details and spatial relationships. The combination of the two provides a more comprehensive scene representation for the model. When one modality is interfered with by noise (such as instantaneous strong light affecting the light intensity signal), the other modality can provide supplementation and correction, greatly improving the system's anti-interference capability. Furthermore, using sequences rather than single frames as input, its backbone network performs feature encoding and correlation calculations on the dual-modal sequences, essentially modeling dynamic information such as pedestrian movement and posture changes in the temporal dimension. This enables the system not only to detect static presence but also to perceive movement trends and micro-motion features, which is beneficial for distinguishing dynamic pedestrians from static background interference and improving the detection capability of partially occluded targets. Furthermore, the backbone network does not process the two types of data independently, but rather fuses them at a deep level through correlation computation (such as attention mechanisms and feature cross-referencing). This enables the target detection model to learn the physical-signal correlation between which modulation pattern corresponds to which light intensity response mode, thereby generating a correlation feature map that encodes the inherent laws of the interaction between the target (pedestrian) and the modulated light field. From the input of the dual-modal sequence to the output of the final detection result, the entire process is completed within a single target detection model. This enables the backbone network to automatically learn the feature representation and correlation method that is most beneficial to the final detection task, achieving global optimization and end-to-end joint optimization, thus improving overall performance. Moreover, it automatically extracts multi-level and highly discriminative features from high-dimensional and complex original sequence data, effectively addressing the diversity of pedestrian appearances (such as different clothing and body types) and the complexity of scenes. It can more accurately identify pedestrians in various challenging scenarios (lighting changes, dynamic backgrounds, partial occlusion), while reducing false alarms and false negatives, and improving the efficiency and accuracy of pedestrian detection in adverse weather conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of an embodiment of the pedestrian detection method based on severe weather provided by the present invention; Figure 2 For the present invention Figure 1 A schematic diagram of an embodiment of S101; Figure 3This is a schematic diagram of the imaging acquisition system of the present invention; Figure 4 This is a schematic diagram of the backbone network structure of the target detection model of the present invention; Figure 5 For the present invention Figure 1 A schematic diagram of an embodiment of S102; Figure 6 This is a schematic diagram of the target detection model of the present invention; Figure 7 This is a schematic flowchart of another embodiment of the pedestrian detection method based on severe weather provided by the present invention; Figure 8 This is a schematic flowchart of another embodiment of the pedestrian detection method based on severe weather provided by the present invention; Figure 9 A schematic diagram of an embodiment of the pedestrian detection system based on severe weather provided by the present invention; Figure 10 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0021] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention provides a pedestrian detection method, system, device, and medium based on severe weather conditions, which are described below.

[0024] Figure 1 This is a schematic flowchart of an embodiment of the pedestrian detection method based on severe weather provided by the present invention, as shown below. Figure 1 As shown, pedestrian detection methods based on severe weather conditions include: S101. Obtain the modulation light field sequence and the corresponding light intensity signal sequence; the modulation light field sequence is generated based on multiple sequentially arranged speckle patterns, the modulation light field sequence includes multiple modulation patterns, and the light intensity signal sequence is time-synchronized with the modulation light field sequence.

[0025] It should be noted that the following embodiments obtain a modulated light field sequence generated by a series of speckle patterns arranged in a specific order, and a corresponding time-synchronized light intensity signal sequence. The modulated light field sequence consists of multiple consecutive modulation patterns, each generated sequentially based on the aforementioned speckle patterns. Simultaneously, the light intensity signal sequence is strictly synchronized with the modulated light field sequence in the time dimension, ensuring that the light intensity signal at each moment is correlated with the corresponding modulation pattern in real time.

[0026] S102. Input the modulation pattern of the modulation light field sequence and the light intensity signal sequence into the target detection model, and use the backbone network of the target detection model to perform feature encoding and correlation calculation on the two sequences to obtain a correlation feature map that represents the correlation information of the light field.

[0027] It should be noted that each modulation pattern in the modulated light field sequence and its time-synchronized light intensity signal sequence are input into the target detection model. The backbone network of the target detection model first performs independent feature encoding on these two heterogeneous sequences, and then performs deep fusion on the encoded features to finally generate a correlation feature map that can comprehensively represent the complex correlation information between the light field and light intensity.

[0028] S103. Based on the target detection model, obtain the pedestrian detection results within the target area according to the associated feature map.

[0029] It should be noted that the object detection model performs decoding analysis based on the generated associated feature map, and finally outputs accurate pedestrian detection results within the target area. The pedestrian detection results include information such as its location, prediction box, and confidence score.

[0030] In summary, the pedestrian detection method based on severe weather provided by this invention first constructs a complementary dual-modal data stream based on synchronously acquired light intensity signal sequences (which can be considered as changes in overall reflection intensity or average brightness) and modulation patterns (containing fine spatial structure modulation information). The light intensity signal is sensitive to rapid changes and large targets, while the modulated light field encodes details and spatial relationships. The combination of these two provides the model with a more comprehensive scene representation. When one modality is affected by noise (such as instantaneous strong light affecting the light intensity signal), the other modality can provide supplementation and correction, greatly improving the system's anti-interference capability. Furthermore, using sequences rather than single frames as input, its backbone network performs feature encoding and correlation calculations on the dual-modal sequences, essentially modeling dynamic information such as pedestrian movement and posture changes in the temporal dimension. This enables the system not only to detect static presence but also to perceive movement trends and micro-motion features, facilitating the differentiation between dynamic pedestrians and static background interference and improving the detection capability of partially occluded targets. Furthermore, the backbone network does not process the two types of data independently, but rather fuses them at a deep level through correlation computation (such as attention mechanisms and feature cross-referencing). This enables the target detection model to learn the physical-signal correlation between which modulation pattern corresponds to which light intensity response mode, thereby generating a correlation feature map that encodes the inherent laws of the interaction between the target (pedestrian) and the modulated light field. From the input of the dual-modal sequence to the output of the final detection result, the entire process is completed within a single target detection model. This allows the backbone network to automatically learn the feature representation and correlation method most beneficial to the final detection task, achieving global optimization and end-to-end joint optimization, thus improving overall performance. Moreover, it automatically extracts multi-level and highly discriminative features from high-dimensional and complex original sequence data, effectively addressing the diversity of pedestrian appearances (such as different clothing and body types) and the complexity of scenes. It can more accurately identify pedestrians in various challenging scenarios (lighting changes, dynamic backgrounds, partial occlusion), while reducing false alarms and false negatives, and improving the efficiency and accuracy of pedestrian detection in adverse weather conditions.

[0031] In environments with scattering media such as fog and haze, to avoid severe degradation of image contrast and low signal-to-noise ratio due to light scattering, thus failing to effectively acquire target features, and to improve the efficiency and accuracy of pedestrian detection in adverse weather conditions, in some embodiments of the present invention, such as Figure 2 As shown, step S101 includes: S201, A light beam is generated and emitted to a spatial light modulator by a light source.

[0032] It should be noted that: Figure 3 This is a schematic diagram of the imaging acquisition system, such as... Figure 3As shown, the imaging acquisition system includes components such as a spatial light modulator 11, a light source 12 (LS), a photodetector 13, and a lens 15. The spatial light modulator 11 is preferably a digital micromirror device (DMD), and the photodetector 13 is preferably a single-pixel bucket detector (BD), such as a silicon photodetector. A 632.8 nm helium-neon laser can be used as the light source 12. The wavelength of this helium-neon laser is in the visible red light band, which has a certain penetrating ability in haze, and the high coherence and monochromaticity of the laser are beneficial for subsequent structured light modulation and detection. The continuous laser beam generated by the helium-neon laser, after beam expansion and collimation, forms a parallel beam with a diameter matching the working surface of the subsequent spatial light modulator 11, and is perpendicularly incident on the modulation plane of the spatial light modulator 11.

[0033] S202. Control the spatial light modulator to generate the speckle pattern, and spatially modulate the received light beam according to the speckle pattern to generate the corresponding modulation pattern in sequence to obtain the modulation light field sequence and project it onto the target area.

[0034] It should be noted that: the main control computer pre-stores a speckle pattern sequence consisting of M speckle patterns, and the system sequentially loads the driving signal of the m-th speckle pattern onto the spatial light modulator 11. Each speckle pattern corresponds to a specific on / off (+12° / -12°) state distribution of a large number of micromirror units on the spatial light modulator 11. The incident parallel laser beam 14 first illuminates the spatial light modulator 11, which performs spatial intensity modulation according to the loaded speckle pattern to form a structured modulation light field (i.e., according to the loaded speckle pattern, micromirrors in the "on" state reflect light to the working optical path, while micromirrors in the "off" state reflect light to the optical trap for absorption. Therefore, the spatial intensity distribution of the reflected beam is "sculpted" in real time into a structured light field that is completely consistent with the currently loaded speckle pattern, thus generating a modulation pattern). The beam corresponding to the modulation pattern passes through the haze medium and is projected onto the target area 20. The control computer cyclically loads the entire preset speckle pattern sequence. Accordingly, the spatial light modulator 11 sequentially generates a set of modulation patterns. , ,..., These time-discrete, sequentially generated light fields collectively constitute the modulated light field sequence. The beam corresponding to this modulated light field sequence is collimated and expanded after passing through the projection lens group, covering the entire target area to be detected in the hazy environment.

[0035] S203. The light signal reflected by the target area is collected by a photodetector to obtain the light intensity signal sequence.

[0036] It should be noted that the light signal reflected from the target area (which may include pedestrians), carrying spatial information about the target, passes through the haze medium again and is received and focused by a large-aperture collecting lens. The function of this lens is to maximize the collection of weak, diffuse signal light. The light signal reflected from the target area 20 (which may include pedestrians), carrying spatial information about the target, passes through the haze medium again and is received and focused by a large-aperture collecting lens 15. The function of this lens 15 is to maximize the collection of weak, diffuse signal light. All the focused light signals are then guided to the photosensitive surface of a photodetector 13. The key characteristic of the photodetector 13 is that it has no spatial resolution capability; the magnitude of its output photocurrent is proportional to the instantaneous total luminous flux (i.e., total optical power) received on the entire photosensitive surface. Therefore, when the m-th modulation pattern... When the corresponding light beam illuminates the target area, the photodetector 13 measures and outputs a corresponding analog electrical signal value of the total light intensity. The system generates a random modulation pattern through a spatial light modulator 11 at the transmitting end to produce a corresponding light beam illuminating the target area in the hazy environment. At the receiving end, a photodetector 13 collects the light intensity signal after atmospheric scattering. The modulated light field sequence and the light intensity signal sequence are used as inputs to the target detection model. (Total light intensity value) Given by the following formula: ; in, Indicates the number of measurements. Indicates the first The field strength distribution of the next step. It is a calculated two-dimensional pattern of field distribution (i.e., speckle pattern) used to illuminate the object. The total light intensity collected by the photodetector 13 is correlated with the corresponding illumination pattern.

[0037] The output signal of photodetector 13 is amplified and converted from analog to digital, and then recorded as a digital sequence to obtain the light intensity signal sequence. , ,..., The system ensures that each... The acquisition time window and the corresponding The projection time windows are completely synchronized, resulting in a time-perfectly one-to-one sequence of integrated light intensity signals. In summary, during system operation, the computer-controlled spatial light modulator 11 sequentially loads a series of pre-designed speckle patterns (e.g., orthogonal speckles generated based on the Hadamard matrix). The continuous laser beam emitted by the light source 12 is modulated by these speckle patterns, forming a sequentially varying sequence of modulated light fields, which is then projected onto the target area filled with haze. The light signal reflected from the target, carrying target information and scattered by the atmosphere, is collected and converged by the lens 15 and received by the photodetector 13. The photodetector 13 does not record the spatial distribution; it only outputs a current or voltage value proportional to the instantaneous total incident light power. Therefore, for each projected modulation pattern, the photodetector 13 synchronously records a corresponding total light intensity value. Ultimately, the system outputs a time-perfectly aligned sequence of modulated light fields and light intensity signals, providing raw data that directly resists scattering interference at the physical level.

[0038] In this embodiment, the received light beam is spatially modulated according to the speckle pattern to generate a modulated light field sequence by sequentially generating the corresponding modulation pattern. By actively projecting the modulated light field sequence, even if the light path is scattered by haze, the interaction mode between the light field and the target is determined. The photodetector 13 is not sensitive to pixel-level spatial distortion caused by random scattering. The light intensity signal sequence is obtained by collecting the light signal reflected after passing through the target area through the photodetector 13. Scattering noise is significantly suppressed from the root of data generation, and the obtained original correlated data pair (modulation pattern) is obtained. Total light intensity value It possesses an inherently high signal-to-noise ratio, thus actively overcoming scattering interference. Furthermore, the acquired data is not a traditional image, but a pair of time series, allowing the target detection model to skip the intermediate step of reconstructing blurred images and directly learn the mapping from interference-resistant physical correlation signals to target semantics, improving target detection accuracy. Moreover, based on the photodetector 13, light modulator 11, light source 12, and lens 15, reliability is improved. Simply replacing the light source (such as an infrared laser) easily extends the system's operating wavelength to a spectral region beyond visible light, achieving all-weather, full-band detection capabilities and broadening its application scenarios.

[0039] How to extract a correlation feature map that can characterize the essence of the target and is more compatible with subsequent target detection models, so as to improve the efficiency and accuracy of pedestrian detection? In some embodiments of the present invention, such as... Figure 4 As shown, the backbone network includes a first coding network and a second coding network, such as... Figure 5 As shown, step S102 includes: S501. Based on the first coding network, feature encoding is performed on each modulation pattern in the modulation light field sequence to obtain a first feature sequence.

[0040] It should be noted that the backbone network of the object detection model (RT-CGIDETR) is executed. The backbone network consists of a two-branch structure of a first encoding network (such as a lightweight ResNet convolutional neural network) and a second encoding network (such as a multilayer perceptron MLP), as well as an association computation module. To improve training stability and model convergence speed, the original input sequence is preprocessed before feature encoding. Specifically, the mean is removed from each modulation pattern in the modulated light field sequence and each light intensity value in the light intensity signal sequence to eliminate the DC background component and improve the numerical stability and effectiveness of subsequent processing. For the modulated light field sequence, the average intensity of all modulation patterns at each spatial position (x, y) is calculated. Then, the mean-removed pattern corresponding to each modulation pattern is obtained. Similarly, for a sequence of light intensity signals, calculate its sequence mean. And obtain the average light intensity corresponding to each light intensity value. Based on the following formula This can separate the signal from the background, eliminate the DC bias of the data, and allow the network to focus more on changing information (i.e., the signal portion relevant to the target). This indicates taking the average.

[0041] The mean-removed modulation pattern sequence { The input is fed into the first encoding network. The first encoding network is a convolutional neural network based on the ResNet architecture, containing multiple convolutional layers, batch normalization layers, and activation functions. Its function is to extract the spatial structure and texture features of each modulation pattern. The first encoding network processes each mean-reduced modulation pattern in the input sequence... Perform forward propagation independently and output a corresponding feature map with higher-dimensional semantic representation. As the first feature, the set of all M output first features constitutes the first feature sequence { At this time, each It is no longer the original light intensity distribution, but contains the depth characteristics of how the lighting mode illuminates the scene.

[0042] S502. Based on the second coding network, feature encoding is performed on each light intensity value in the light intensity signal sequence to obtain a second feature sequence.

[0043] It should be noted that: the light intensity value sequence after removing the mean { The input is fed into the second encoding network. The second encoding network is a fully connected network whose function is to map one-dimensional scalar light intensity values ​​to a high-dimensional feature space that matches the spatial features. The second encoding network processes each light intensity value... As input, it undergoes nonlinear transformation through several fully connected layers, outputting a feature vector. As the second feature. This feature vector The dimensions are designed to be semantically and dimensionally compatible with the feature map output by the first encoding network. Effective interaction is then performed in subsequent steps. All M feature vectors The set of constitutes the second feature sequence { }

[0044] S503. The associated feature map is calculated based on the first feature sequence and the second feature sequence.

[0045] It should be noted that the association calculation module receives the above two feature sequences (including the first feature sequence { } and second feature sequence { }), and each one-dimensional feature vector in the second feature sequence Extending the spatial dimension generates a two-dimensional feature map where the value at each spatial location is the same, equal to This ensures that it can be compared with the two-dimensional feature map in the first feature sequence. Perform element-wise operations. Element-wise multiplication and sequence averaging: for each pair of time-aligned feature maps... and the expanded First, perform element-wise multiplication to obtain an intermediate result. Then, sum and average the intermediate results of all M measurements. The mathematical expression of this operation is: ,in, This indicates element-wise multiplication. This indicates the space expansion operation described above.

[0046] Finally, a single-channel, two-dimensional correlation feature map is calculated. This joint feature map is a deep feature-level fusion and compression of M active measurement information.

[0047] In this embodiment, the backbone network of the target detection model adopts an improved dual-branch coding structure. This structure processes two sequences in parallel: the first coding network performs convolution and other operations on each modulation pattern in the modulation light field sequence to extract its spatial structure features and outputs a first feature sequence. The second coding network performs a fully connected mapping on each light intensity value in the light intensity signal sequence, upscaling it to a semantic space that matches the spatial features and outputs a second feature sequence. Subsequently, the backbone network simulates the physical correlation process of computer imaging, fusing and averaging the two feature sequences to obtain a single-channel two-dimensional correlation feature map. This effectively suppresses random noise, and its signal-to-noise ratio is significantly higher than that of images directly captured under haze conditions. Furthermore, the first and second coding networks upscal the original data to a high-dimensional feature space. The correlation operations performed in this space not only inherit the noise resistance of physical correlation but also incorporate rich semantic information learned by the neural network, resulting in a more refined final output correlation feature map. It is robust to haze scattering and rich in high-level features that are beneficial for target detection. It provides a standard, static input format for subsequent target detection models (such as Transformer and detection head), which allows for direct processing using mature and efficient image target detection architectures without the need to design complex temporal models. This reduces model design complexity, improves computational efficiency, and compresses high-dimensional temporal information into static feature maps, greatly simplifying subsequent processing.

[0048] In complex environments such as smog, the multi-scale target information contained in the associated feature map (especially the details of small distant targets and the semantics of large nearby targets) is difficult to effectively integrate and utilize, resulting in insufficient perception and localization accuracy of target detection models for pedestrians (especially small targets) at different scales. To address this issue, in some embodiments of the present invention, such as... Figure 4 and Figure 6 As shown, the backbone network further includes a convolutional neural network module, and the object detection model further includes a hybrid encoder connected to the output of the convolutional neural network module. The hybrid encoder includes an adaptive interleaved feature interaction module and a cross-scale channel fusion module; as shown... Figure 7 As shown, the method further includes: S701. Based on the convolutional neural network module, feature encoding and multi-scale downsampling are performed on the associated feature map to output multiple initial feature maps of different scales.

[0049] It should be noted that the convolutional neural network module in the backbone network receives the single-channel association feature map generated by the previous association calculation step. As input, the convolutional neural network module is a standard downsampling convolutional network containing multiple stages (e.g., later stages based on ResNet), and its workflow is as follows: First, the associated feature maps are processed through several convolutional layers. Perform nonlinear transformation and channel expansion to correlate feature maps The process of converting a single-channel feature map into a deep feature map with multiple channels (e.g., 32 or 64 channels) enriches the semantic information of the features. Then, several regular spatial downsampling operations are performed through stride convolution or pooling. After each downsampling, the spatial dimensions (height and width) of the deep feature map are halved, while the number of channels typically doubles, progressively increasing the level of semantic abstraction. After a predetermined number of downsampling operations, a set of three (or more) initial feature maps at different scales is output. For example, the three initial feature maps at different scales are typically labeled S3, S4, and S5. Initial feature map S3 has the largest scale and highest resolution, preserving the richest spatial details and edge information, which is crucial for locating small targets. Initial feature map S4 is of medium scale, balancing some spatial details with higher semantic information. Initial feature map S5 has the smallest scale and lowest resolution, but its semantic information is the most abstract and global, containing high-level information such as the target's category and overall contour.

[0050] S702. Based on the adaptive interleaved feature interaction module, the initial feature map with the smallest scale is serialized to obtain the target feature.

[0051] It should be noted that the Adaptive Interleaved Feature Interaction (AIFI) module within the hybrid encoder is essentially a Transformer encoder. The AIFI module receives the initial feature map S5, which has the highest semantic level and smallest scale, as input. Because the initial feature map S5 undergoes the deepest convolutional processing, its spatial size is the smallest, but the feature vector at each location contains the richest global semantic information, making it most suitable for global context modeling by the Transformer. First, the two-dimensional S5 feature map (shape [C, H, W]) is unfolded spatially, reshaping it into a one-dimensional feature sequence (shape [L, C], where L = H × W), where each sequence element can be considered a high-level visual vocabulary. This one-dimensional feature sequence is then input into a multi-layer Transformer encoder. Through its core self-attention mechanism, each element in the one-dimensional feature sequence can interact and perform weight calculations with all other elements in the sequence, effectively establishing long-range dependencies between any two distant regions in the associated feature map. For example, associating a pedestrian's head features with their footsteps, or distinguishing a target from a blurred background context in smog. After multiple layers of interaction, each element in the feature sequence incorporates global contextual information. After fully modeling the global context through multiple Transformer blocks, the processed one-dimensional feature sequence is readjusted back into a two-dimensional spatial structure (with the same shape as S5), resulting in the refined target feature F5. At this point, the target feature F5 not only inherits the rich high-level semantic information of S5 but also incorporates a deeper understanding of the global context, providing a higher-quality feature representation for subsequent cross-scale fusion.

[0052] S703. Based on the cross-scale channel fusion module, the target features are fused with initial feature maps of other scales to generate multi-scale fused features.

[0053] It should be noted that the cross-scale channel fusion module within the hybrid encoder acquires the refined target feature F5 (with global context) from the AIFI module, as well as initial feature maps from other scales (e.g., the original initial feature maps S3 and S4 with rich spatial details) from the backbone network. Since the initial feature maps at other scales (such as S3 and S4), the target feature F5, and the target feature itself have different channel numbers and resolutions, they are first adjusted to a consistent intermediate channel number and a suitable resolution for fusion (usually based on the resolution of S4) through 1×1 convolutions or upsampling / downsampling operations. Then, an attention mechanism (such as channel attention) is used to assign appropriate weights to the feature channels from different scales. For example, when precise localization is required, higher weights are assigned to the S3 feature, which contains more edge details; when class determination is required, higher weights are assigned to the F5 feature, which is rich in global semantics. Another implementation is a simple weighted summation, but the weights are learned by the network. After the above adaptive fusion operation, a unified multi-scale fused feature is output. The multi-scale fusion feature includes fine spatial details from S3 to facilitate small target localization and bounding box regression, as well as intermediate layer information from S4. More importantly, it deeply embeds high-level semantic information from F5, which has been enhanced with global context, to facilitate target classification and resist background interference.

[0054] In this embodiment, the correlation feature map output by the backbone network First, a convolutional module performs multi-scale downsampling to obtain a set of initial feature maps at multiple scales (e.g., S3, S4, S5). These initial feature maps are then fed into a hybrid encoder. The hybrid encoder first performs global context modeling on the most abstract feature maps through an adaptive interleaved feature interaction module (a Transformer encoder) to capture long-distance dependencies; then, a cross-scale channel fusion module adaptively fuses features at all scales to generate a unified and information-rich multi-scale fused feature. By explicitly generating and preserving the high-resolution initial feature map S3 (S701), and adaptively combining it with high-level semantic features in the fusion stage (S703), pixel-level detail information of small targets (such as distant pedestrians) can be directly transmitted to the detection head, thereby significantly reducing the false negative rate of small targets and significantly improving the detection accuracy of small and multi-scale targets, thus improving the overall detection accuracy. Furthermore, in complex scenes such as fog and haze, local features are prone to ambiguity due to occlusion or blurring. The AIFI module (S702) utilizes the Transformer's self-attention mechanism to understand and enhance local features based on the overall scene layout (e.g., using road structure information to infer the possible location of pedestrians). This effectively suppresses false detections caused by local blurring or occlusion, improving the model's robustness and accuracy in complex backgrounds. Furthermore, the CCFF module (S703) dynamically determines how much and what information to extract from features at different scales based on the specific content of the current input. This ensures that the final multi-scale fused features achieve an optimal balance between details and semantics, and between local and global information, providing the highest quality feature input for subsequent detection tasks. This enables accurate pedestrian detection and improves detection robustness.

[0055] To further improve the accuracy and efficiency of pedestrian recognition in complex scenarios such as smog, in some embodiments of the present invention, such as... Figure 6 As shown, the target detection model also includes a query selector, a decoder, and a detection head; as Figure 8 As shown, the method further includes: S801. The multi-scale fusion features are interacted with a set of preset initial query vectors through attention to obtain the preliminary prediction score corresponding to each initial query vector.

[0056] It should be noted that a set of initial query vectors (e.g., 300) are preset. These initial query vectors are learnable parameters in the object detection model, essentially a set of high-dimensional vectors. Multi-scale fusion features from S703 are used as keys and values, and the initial query vectors are used as queries, input into a multi-head cross-attention module. This module allows each initial query vector to adaptively collect relevant information from all spatial locations and semantic channels of the multi-scale fusion features. After interaction, each initial query vector is fed into a lightweight preliminary detection head (typically composed of a few fully connected layers). This head outputs a preliminary prediction score for each initial query vector. The preliminary prediction score includes confidence, the predicted bounding box, and its corresponding localization score (e.g., Intersection over Union (IoU)). The confidence is a probability distribution across all candidate categories (e.g., "pedestrian" or "background"), representing the probability that the query currently considers the target to belong to each category. The preliminary prediction score may also include the IoU score of the predicted bounding box and a reference bounding box (the ground truth bounding box), used to measure the initial quality of the localization.

[0057] S802. The query selector calculates the uncertainty score of each initial query vector based on the preliminary prediction score, and selects multiple candidate query vectors with the lowest uncertainty scores from the initial query vectors.

[0058] It should be noted that the query selector calculates an uncertainty score for each initial query based on the preliminary prediction score obtained from S801. If a query's preliminary prediction score shows a very high probability of corresponding to "background" and its predicted bounding box's IoU score is extremely low, it indicates that the query is in a vague and meaningless region, and its prediction result is unreliable with high uncertainty. Conversely, if a query has high confidence in the "pedestrian" category and high quality (IoU) of the predicted bounding box, its uncertainty is low and its information value is high. The query selector uses a comprehensive scoring function to quantify the "uncertainty" or "information value" of each query. The comprehensive scoring function calculates an uncertainty score from the confidence score and the location score, and is designed so that a higher uncertainty score indicates lower uncertainty and higher information value. For example, the uncertainty score S = α · pobj + β IoU, among which, α , β The preset weighting coefficients, pobj Here, IoU represents the confidence level, and the Intersection over Union (IoU) score represents the score. All initial query vectors are sorted by their uncertainty scores, and the K (e.g., 30) initial query vectors with the lowest uncertainty scores are selected as candidate query vectors. This allows computational resources to be focused on the regions where the target object (e.g., pedestrians) is most likely to be present.

[0059] S803. Based on the decoder, the candidate query vector is decoded and refined to obtain the target query vector.

[0060] It's important to note that the decoder typically consists of multiple Transformer decoding layers. The decoder receives candidate query vectors selected by the query selector and multi-scale fused features from the hybrid encoder. In each layer of the decoder, candidate query vectors first interact with each other through a self-attention mechanism. This process enables each query to perceive the content and state of the others, promoting task division and preventing multiple query vectors from repeatedly responding to the same target. Next, the candidate query vectors interact with the multi-scale fused features through a cross-attention mechanism, allowing each candidate query vector to extract the most relevant semantic and locational information from the image features, thereby refining the category and bounding box of the target it represents. This process is repeated layer by layer in the multi-layer decoder. Each layer further integrates collaborative information between queries and contextual information from the image, building upon the output of the previous layer. After multiple iterations, the initial candidate query vectors are gradually refined into more discriminative target query vectors, each vector corresponding to an independent potential target instance with more clearly defined localization and semantic information. Finally, the target query vector output from the last layer decoder will be fed into the detection head (usually a fully connected network) to directly generate the class probability and bounding box coordinates of each target, completing the end-to-end detection output.

[0061] S804. Based on the detection head and the target query vector, output the probability distribution of the pedestrian's category and the coordinates of the predicted bounding box representing the pedestrian's location, thereby generating the pedestrian detection result.

[0062] It's important to note that the detection head typically consists of two parallel fully connected networks. The detection head includes a classification branch and a regression branch. One fully connected network, acting as the classification branch, receives the target query vector and outputs the final class probability distribution (usually normalized using a softmax function), determining whether the target query vector corresponds to "pedestrian" or another category (such as background). The other fully connected network, acting as the regression branch, receives the same target query vector and outputs a set of normalized bounding box coordinates (usually center point coordinates, width, and height). These coordinates directly correspond to the pedestrian's position in the image. Each retained target query vector, after passing through the detection head, produces a complete detection result (class + bounding box). Because the S802's query filtering mechanism efficiently removes a large number of redundant and low-quality initial query vectors, the detection head outputs a highly concise, almost non-overlapping set of detection results, eliminating the need for post-processing operations such as non-maximum suppression as the final detection result.

[0063] In this embodiment, a query selector (S802) dynamically filters query vectors before decoding, performing only expensive refinement and prediction calculations on a small number of high-value initial query vectors (i.e., the K with the lowest uncertainty scores). This significantly reduces the computational cost (FLOPs) and memory usage of the forward propagation of the object detection model, enabling real-time performance of the object detection model in embedded devices or scenarios requiring high frame rate responses (such as autonomous driving), thus improving detection efficiency. Furthermore, under low-quality inputs such as smog, many queries become stuck in blurred background areas, generating meaningless predictions and leading to a high false positive rate. The uncertainty-based filtering mechanism is essentially a highly efficient noise filter, actively eliminating initial query vectors that perform poorly in the initial judgment and correspond to background or blurred areas. This prevents these "noise queries" from being further amplified in the subsequent decoding process, resulting in erroneous outputs. This allows the object detection model to focus its "attention" and modeling capabilities on genuine potential targets, thereby improving overall detection accuracy, especially reducing the false positive rate. Furthermore, the entire process (S801-S804) forms a perfect closed loop: through interaction → evaluation → screening → refinement → output, this screening mechanism, along with the decoding and detection head, is optimized in terms of quantity, thus completely eliminating the post-processing stage that relies on manual threshold tuning and non-maximum suppression. This not only further accelerates the inference speed but also eliminates the performance degradation caused by improper post-processing parameter settings, making the entire system more robust and efficient, and improving the accuracy and efficiency of pedestrian detection.

[0064] To train a target detection model that accurately identifies pedestrians in complex scenarios such as fog and haze, some embodiments of the present invention further include: A training dataset is constructed, which contains multiple sets of sample data collected under simulated or real severe weather conditions. Each set of sample data includes a modulated light field sequence, a time-synchronized integrated light intensity signal sequence, and corresponding labels indicating the pedestrian target category and location.

[0065] It should be noted that sample data must be collected under simulated or real hazy weather conditions. Simulated environments can be achieved in a laboratory using a fog chamber to precisely control haze concentration, while real environments are conducted outdoors on actual hazy days to ensure the authenticity of the data distribution. Each training sample is a triplet (modulated light field sequence, integrated light intensity signal sequence, label). The modulated light field sequence and light intensity signal sequence are recorded from known speckle pattern sequences projected sequentially by the DMD, based on the above embodiments S201 to S203. Accurate pedestrian information is labeled to obtain labels corresponding to the current scene. Labels typically include a category label for each pedestrian (e.g., "pedestrian") and the predicted bounding box position (represented in [x_min, y_min, x_max, y_max] or [center x, center y, width, height] format). All collected sample data are divided into training, validation, and test sets according to a preset ratio (e.g., 8:1:1). To improve model robustness, slight light intensity fluctuations can be applied to the sequence data during training to simulate noise, or the label boxes can be slightly randomly translated and scaled as data augmentation.

[0066] With the goal of minimizing the prediction error, the initial model is iteratively trained end-to-end using the training dataset to obtain the object detection model; wherein, the prediction error is calculated based on the classification loss and the bounding box regression loss.

[0067] It should be noted that the training objective is to optimize an initial model with an RT-CGIDETR architecture (with randomly initialized parameters) into a pedestrian detection model that can accurately detect pedestrians in hazy scenes through supervised training. Deep learning frameworks such as PyTorch or TensorFlow are used, and the optimizer chosen is AdamW, which combines adaptive learning rate and weight decay (set to 1e). 5) It can effectively stabilize the training of Transformer-type models and suppress overfitting. The initial learning rate is set to 5e. 3. The ReduceLROnPlateau scheduler is used to dynamically adjust the learning rate: when the validation set performance (e.g., mAP) does not improve within 10 consecutive epochs, the learning rate is reduced by multiplying it by a decay factor of 0.1 until it is no lower than 1e. 6. This strategy enables fine-tuning of the model in the later stages of training, promoting stable convergence. A multi-task loss function is used to calculate the prediction error, which is a weighted sum of the classification loss and the bounding box regression loss. The classification loss is typically calculated using Focal Loss or cross-entropy loss, measuring the difference between the model's predicted class probability distribution and the true class, primarily optimizing target recognition capabilities. The bounding box regression loss is typically calculated using GIoULoss or L1 loss, measuring the difference between the model's predicted bounding box location and the true bounding box location, primarily optimizing target localization accuracy. The entire RT-CGIDETR architecture's initial model (from the backbone network to the detection head) is trained as a whole in an end-to-end manner. Gradients are backpropagated from the final detection loss to the front end of the network, achieving global parameter optimization. On high-performance GPUs such as the NVIDIA RTX 3080, approximately 256 training iterations are typically required to ensure sufficient loss convergence and achieve stable peak performance on the validation set. These configurations collectively ensure that the model can efficiently and stably learn robust pedestrian detection capabilities in hazy environments from a randomly initialized state.

[0068] In this embodiment, raw associated sequence data is directly collected as the training set in a real or high-fidelity simulated haze environment. This allows the model to be exposed to and adapt to the data feature distribution caused by haze scattering from the very beginning of learning, avoiding the domain gap problem caused by training with clear images and then testing with haze images. This ensures that the trained target detection model has inherent and optimal adaptability and robustness to severe weather scenarios. Furthermore, an end-to-end training approach is adopted, placing the unique backbone network (processing CGI data), efficient encoder, intelligent query selector, and detection head under the same optimization objective. This allows the front-end feature extraction network to learn the feature representation most suitable for the final detection task, while the back-end detection decision module can also be optimized based on the most beneficial features. With the goal of minimizing prediction error, an advanced training strategy combining the AdamW optimizer with dynamic learning rate scheduling is used to effectively manage the complexity and stability of the training process, avoiding gradient explosion / vanishing and promoting model convergence. The balanced design of the multi-task loss function ensures the simultaneous improvement of model accuracy and localization capability.

[0069] To improve imaging and detection efficiency, in some embodiments of the present invention, the speckle pattern is generated based on the Hadamard matrix.

[0070] It's important to note that the deterministic orthogonal structure of the Hadamard matrix is ​​used to modulate the optical field. First, an N-order Hadamard matrix is ​​generated in the control computer, where N is a power of 2 (e.g., 64, 128, 256, etc.). The Hadamard matrix is ​​a square matrix consisting only of +1 and -1 elements, and its most important property is that any two rows (or columns) are mutually orthogonal. In practical applications, M column vectors (or row vectors) of the Hadamard matrix are typically selected as a modulation basis, where M can be less than or equal to N to match the required number of samples. Each selected Hadamard column vector (a one-dimensional sequence of +1 and -1) is reshaped into a two-dimensional array, its size matching the regional resolution used for modulation on the digital micromirror device (DMD) (e.g., reshaped into a 256×256 two-dimensional array). Thus, each column vector corresponds to a two-dimensional "pattern template." The element values ​​"+1" and "-1" in the pattern template are mapped to two physical states of the DMD micromirror. Typically, the mapping relationship is as follows: "+1" corresponds to the first intensity state of the micromirror (reflecting light to the working optical path), and "-1" corresponds to the second intensity state of the micromirror (reflecting light to the light absorber). Through this mapping, a mathematical Hadamard vector is transformed into a physically realizable, binary spatial light intensity distribution map, i.e., a deterministic speckle pattern, thus obtaining a sequence of deterministic speckle patterns with an orthogonal structure composed of N patterns. The deterministic speckle pattern sequence is sequentially loaded into the digital micromirror device (DMD) to control the reflection state of each micromirror unit, thereby generating and projecting a modulated light field sequence with orthogonality corresponding to the pattern sequence. That is, the binary patterns (i.e., speckle patterns) corresponding to M Hadamard column vectors are sequentially loaded into the DMD, and the DMD rapidly refreshes the state of its micromirror array according to each speckle pattern. When the laser beam irradiates the surface of the DMD, it is modulated by these sequentially switched speckle patterns, thereby generating a modulated light field sequence composed of deterministic speckles that varies over time and is projected onto the target area.

[0071] In this embodiment, a Hadamard matrix is ​​used to generate the speckle pattern. Due to its inherent orthogonality, the Hadamard matrix ensures that the information carried by each sample does not overlap, greatly improving the information acquisition efficiency. This allows for the rapid and efficient acquisition of sufficient feature information for pedestrian detection, significantly reducing the data acquisition time required for a single detection and thus greatly improving pedestrian detection efficiency. Furthermore, the speckle pattern generated by the Hadamard matrix possesses strong noise suppression capabilities. Even under extreme weather conditions with low illumination and high noise, the acquired light intensity signal sequence has a higher signal-to-noise ratio, providing a cleaner and more reliable input for subsequent deep learning models and improving the environmental adaptability of pedestrian detection.

[0072] For example, such as Figure 4 and Figure 6As shown, the RT-CGIDETR model includes a backbone network, which first modulates the input optical field... and the collected light intensity signal Feature encoding is performed to obtain and .in ,and .in It is used for encoding The first encoding network of information, It is used for encoding A fully connected information network. Subsequently, association calculations are performed to obtain an association feature map. At this point, we get It is a single-channel two-dimensional data containing complete optical field correlation information (multi-channel data requires multiple detectors and corresponding illumination speckles for acquisition), and its size is similar to... Maintain consistency. Then, use the ResNet network to... Feature encoding yields features with 32 channels. Each channel is overlaid with the original Information, that is This method utilizes deep learning to simulate ghost imaging technology to acquire image information under extreme weather conditions such as smog. It leverages not only the anti-interference capabilities of ghost imaging but also the rapid feature extraction capabilities of deep learning, thus ensuring the method's robustness and real-time performance. Subsequently, the backbone network processes the features... Perform multi-scale feature calculation and extraction to obtain The RT-CGIDETR model performs feature encoding and decoding on the input data and performs object detection. Its core principle is to utilize the RT-CGIDETR model's efficient hybrid encoder (EHE) to improve its ability to process objects of different scales, thereby ensuring accuracy in detecting small targets. The efficient hybrid encoder processes feature maps from the backbone network.

[0073] The Adaptive Interleaved Feature Interaction (AIFI) module is a crucial component of EHE, with its core being a multi-layer Transformer encoder. EHE first uses intra-scale interaction to transform the 2D S5 features into a sequence vector, which is then processed by the AIFI module. The output is then adjusted back to 2D to obtain F5, and S3, S4, and F5 are fed into the Cross-Scale Channel Fusion (CCFF) module. Cross-scale fusion transforms multi-scale features into a series of image features. The CCFF module performs multi-level fusion, combining features from different scales to maintain a balance between high-level semantic information and low-level detail features. The CCFF module typically employs a cross-channel attention mechanism or a simple weighted average to accomplish this task. The lightweight nature of the RT-CGIDETR model hinges on the query selector. This component selects based on the uncertainty of the query. In DETR-type models (including RT-DETR and its derivative RT-CGIDETR), the query vector is the core carrier of the detection task. The query vector is essentially a set of learnable vectors. The model interacts with the feature map output by the encoder through attention interactions using the query vector, ultimately predicting the target's category, bounding box (IoU), and other information from the query vector. The model first allows all initial query vectors to undergo preliminary attention interaction with the features output by the encoder, obtaining preliminary prediction results for each initial query vector: including the probability distribution of the target category (classification score) and the intersection-over-union (IoU) score between the predicted bounding box and the ground truth bounding box. If a query vector has an extremely low classification score (e.g., a 99% probability of background) and an IoU score close to 0, it indicates that the query vector corresponds to a meaningless background region, with high uncertainty (i.e., no value to the detection task). If a query vector has a high classification score (e.g., a 95% probability of "pedestrian") and a high IoU score (e.g., 0.8), it indicates that the query vector accurately matches the target, with low uncertainty (i.e., high information content). The query selector sets a filtering threshold based on the comprehensive score of "classification + IoU score" (or selects the top N by score), retaining initial query vectors with low uncertainty and high information content, and discarding redundant query vectors with high uncertainty. The query selector ensures that the most informative target query vector is selected for the detection task, thereby reducing redundancy and improving efficiency. The decoder and head module of the RT-CGIDETR model process the selected queries to produce the final detection output. The decoder performs positional encoding, image feature refinement, and object query learning on the query vector. The RT-CGIDETR model ultimately uses two types of fully connected networks for detection.

[0074] This invention employs CGI and deep learning technologies to complete data acquisition and target detection tasks, aiming to solve the problem of low target detection accuracy of traditional methods under extreme weather conditions such as smog. CGI adaptively modulates and reconstructs the scattered light field under extreme weather conditions, overcoming the scattering interference bottleneck of traditional optical imaging in smoggy environments. As an imaging technology based on the correlation characteristics of light field, the core advantage of CGI is that it does not rely on direct target imaging, but recovers target information through intensity correlation calculations between reference light and object light. Even in scattering media (smog, rain, fog), it can still remove scattering noise and reconstruct a clear target outline through light field phase compensation algorithms. At the same time, traditional CGI requires tens of thousands of speckle samples to obtain a clear image for detection through correlation calculations, while the end-to-end model in this invention only requires 1 / 20 of the number of speckles to achieve high-accuracy detection. For example, when identifying simple geometric targets (such as MNIST handwritten digits), the sampling rate (SR) can be reduced to 0.05 while still stably outputting detection results, greatly reducing data acquisition and processing time. Since CGI only requires one detector to collect light source information, its physical resources are greatly reduced compared to traditional two-dimensional pixel array imaging. The deep learning in this invention is not only used for target detection but also for optimizing and computing the acquired data. Utilizing the feature extraction capabilities of deep learning methods for imaging computation achieves high-quality and high-efficiency image information recovery. Compared to traditional Faster R-CNN and YOLO, RT-DETR is a lightweight end-to-end target detection network model, with a detection speed 21 times faster than YOLO and accuracy essentially the same. Based on these advantages, the RT-CGIDETR model proposed in this invention achieves a 10% improvement in target detection accuracy (AP) compared to traditional methods in hazy weather conditions.

[0075] This invention employs simulation and experimentation to verify the feasibility of the method. In the simulation, measurements collected from objects illuminated by random patterns are used as the object detection results. Since random speckle patterns align with the physical nature of CGI (light field correlation), are easily generated in simulations and can verify the principle, a Gaussian-distributed random pattern is used as the measurement matrix. Simultaneously, to ensure the repeatability of the experiment, a fixed random seed is maintained. These measurements are then paired with corresponding patterns, and the true values ​​are labeled as the core data material for supervised model training. The entire dataset is divided into training, testing, and validation sets in an 8:1:1 ratio to ensure scientific data partitioning and the reliability of experimental results. To comprehensively evaluate the model's performance limits under different conditions, the model is tested at various sampling rates to analyze the impact of sampling rate variations on model detection performance. Furthermore, the network architecture is implemented using the PyTorch framework, and the model has been systematically trained. To ensure optimal balance among different components, the loss term is assigned equal weights in a 1:1 ratio. The AdamW optimizer was used in the optimization algorithm, with an initial learning rate of 5e-3 and a weight decay factor of 1e-5 to effectively reduce the risk of overfitting and improve the model's generalization ability. The ReduceLROnPlateau scheduler was used to adjust the dynamic learning rate. When there was no significant performance improvement over 10 consecutive epochs, the scheduler reduced the learning rate by a decay factor of 0.1, while ensuring it did not fall below the minimum threshold of 1e-6. The entire training process was completed on an Nvidia RTX 3080 GPU, with a total of 256 training epochs to fully optimize model performance and ensure training stability and convergence. In the experiments, a helium-neon laser with a wavelength of 632.8 nm was used. The laser beam was first irradiated onto a digital micromirror device (DMD; DLP LightCrafter, a 0.3x aspect ratio chipset), and then the modulated light was projected onto an object. The light reflected from the object was collected by a barrel detector (DET36A silicon photodetector). Because the column vectors of the Hadamard matrix are mutually orthogonal, the information carried by each speckle pattern does not overlap, requiring only... Image reconstruction requires only a few samples (N being the number of image pixels), far fewer than the sampling number required for random speckle. Furthermore, random speckle is highly sensitive to detector noise and environmental interference (its fluctuation characteristics are easily masked by noise), while the "deterministic structure" of Hadamard speckle can be noise-suppressed through orthogonal decoding. Therefore, a Hadamard matrix was used to illuminate the object in the experiment. The collected data was then used as input to the model for training and prediction. The network model training settings in the experiment were consistent with the simulation settings.

[0076] To better implement the pedestrian detection method based on severe weather in this invention embodiment, based on the pedestrian detection method based on severe weather, correspondingly, as follows: Figure 9As shown, this embodiment of the invention also provides a pedestrian detection system 900 based on severe weather conditions. The pedestrian detection system 900 based on severe weather conditions includes: The acquisition module 901 is used to acquire a modulated light field sequence and a corresponding light intensity signal sequence; the modulated light field sequence is generated based on multiple sequentially arranged speckle patterns, the modulated light field sequence includes multiple modulation patterns, and the light intensity signal sequence is time-synchronized with the modulated light field sequence; The encoding and calculation module 902 is used to input the modulation pattern and the light intensity signal sequence of the modulation light field sequence into the target detection model, and to perform feature encoding and correlation calculation on the two sequences through the backbone network of the target detection model to obtain a correlation feature map that represents the correlation information of the light field. The detection and recognition module 903 is used to obtain pedestrian detection results within the target area based on the target detection model and the associated feature map.

[0077] The pedestrian detection system 900 based on severe weather provided in the above embodiments can realize the technical solutions described in the above embodiments of pedestrian detection method based on severe weather. The specific implementation principles of each module or unit can be found in the corresponding content in the above embodiments of pedestrian detection method based on severe weather, and will not be repeated here.

[0078] like Figure 10 As shown, the present invention also provides an electronic device 1000. The electronic device 1000 includes a processor 1001, a memory 1002, and a display 1003. Figure 10 Only some components of the electronic device 1000 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0079] In some embodiments, processor 1001 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 1002 or process data, such as the pedestrian detection method based on severe weather in this invention.

[0080] In some embodiments, processor 1001 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 1001 may be local or remote. In some embodiments, processor 1001 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0081] In some embodiments, memory 1002 may be an internal storage unit of electronic device 1000, such as a hard disk or memory of electronic device 1000. In other embodiments, memory 1002 may also be an external storage device of electronic device 1000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 1000.

[0082] Furthermore, the memory 1002 may include both internal storage units of the electronic device 1000 and external storage devices. The memory 1002 is used to store application software and various types of data installed on the electronic device 1000.

[0083] In some embodiments, display 1003 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 1003 is used to display information from electronic device 1000 and to display a visual user interface. Components 1001-1003 of electronic device 1000 communicate with each other via a system bus.

[0084] In one embodiment, when processor 1001 executes a pedestrian detection program based on severe weather in memory 1002, the following steps may be performed: Acquire a modulated light field sequence and a corresponding light intensity signal sequence; the modulated light field sequence is generated based on multiple sequentially arranged speckle patterns, the modulated light field sequence includes multiple modulation patterns, and the light intensity signal sequence is time-synchronized with the modulated light field sequence; The modulation pattern and the light intensity signal sequence of the modulation light field sequence are input into the target detection model. The backbone network of the target detection model performs feature encoding and correlation calculation on the two sequences to obtain a correlation feature map that represents the correlation information of the light field. Based on the target detection model, pedestrian detection results within the target area are obtained according to the associated feature map.

[0085] It should be understood that when the processor 1001 executes the pedestrian detection program based on severe weather in the memory 1002, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0086] Furthermore, the embodiments of the present invention do not specifically limit the type of the electronic device 1000 mentioned. The electronic device 1000 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1000 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0087] Accordingly, this application also provides a computer-readable storage medium for storing a computer-readable program or instruction. When the program or instruction is executed by a processor, it can implement the steps or functions of the pedestrian detection method based on severe weather provided in the above-described method embodiments.

[0088] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0089] The present invention has provided a detailed description of the pedestrian detection method, system, equipment, and medium based on severe weather conditions. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A pedestrian detection method based on severe weather, characterized in that, include: Acquire the modulated optical field sequence and the corresponding optical intensity signal sequence; The modulated light field sequence is generated based on multiple sequentially arranged speckle patterns, the modulated light field sequence includes multiple modulation patterns, and the light intensity signal sequence is time-synchronized with the modulated light field sequence; The modulation pattern and the light intensity signal sequence of the modulation light field sequence are input into the target detection model. The backbone network of the target detection model performs feature encoding and correlation calculation on the two sequences to obtain a correlation feature map that represents the correlation information of the light field. Based on the target detection model, pedestrian detection results within the target area are obtained according to the associated feature map.

2. The pedestrian detection method based on severe weather as described in claim 1, characterized in that, The acquisition of the modulated light field sequence and the corresponding light intensity signal sequence includes: A light beam is generated and emitted by a light source to a spatial light modulator; The spatial light modulator is controlled to generate the speckle pattern, and the received light beam is spatially modulated according to the speckle pattern to generate the corresponding modulation pattern in sequence, thereby obtaining the modulation light field sequence and projecting it onto the target area. The light intensity signal sequence is obtained by collecting the light signal reflected after passing through the target area using a photodetector.

3. The pedestrian detection method based on severe weather as described in claim 1, characterized in that, The backbone network includes a first encoding network and a second encoding network. The backbone network of the target detection model performs feature encoding and association calculation on the two sequences to obtain an association feature map representing the light field association information, including: Based on the first coding network, feature encoding is performed on each modulation pattern in the modulation light field sequence to obtain a first feature sequence; Based on the second coding network, feature encoding is performed on each light intensity value in the light intensity signal sequence to obtain a second feature sequence; The associated feature map is calculated based on the first feature sequence and the second feature sequence.

4. The pedestrian detection method based on severe weather as described in claim 1, characterized in that, The backbone network further includes a convolutional neural network module, and the object detection model further includes a hybrid encoder connected to the output of the convolutional neural network module. The hybrid encoder includes an adaptive interleaved feature interaction module and a cross-scale channel fusion module; the method further includes: Based on the convolutional neural network module, feature encoding and multi-scale downsampling are performed on the associated feature map to output multiple initial feature maps of different scales; Based on the adaptive interleaved feature interaction module, the initial feature map with the smallest scale is serialized to obtain the target feature; The target features are fused with initial feature maps of other scales based on the cross-scale channel fusion module to generate multi-scale fused features.

5. The pedestrian detection method based on severe weather as described in claim 4, characterized in that, The target detection model further includes a query selector, a decoder, and a detection head; the method further includes: The multi-scale fusion features are interacted with a set of preset initial query vectors through attention to obtain a preliminary prediction score corresponding to each initial query vector; The query selector calculates the uncertainty score of each initial query vector based on the preliminary prediction score, and selects multiple candidate query vectors with the lowest uncertainty scores from the initial query vectors. The candidate query vector is decoded and refined based on the decoder to obtain the target query vector; Based on the detection head and the target query vector, the probability distribution of the pedestrian's category and the coordinates of the predicted bounding box representing the pedestrian's location are output, thereby generating the pedestrian detection result.

6. The pedestrian detection method based on severe weather as described in claim 1, characterized in that, Also includes: Construct a training dataset, which contains multiple sets of sample data collected under simulated or real severe weather conditions. Each set of sample data includes a modulated light field sequence, a time-synchronized integrated light intensity signal sequence, and corresponding labels labeled with pedestrian target categories and locations. With the goal of minimizing the prediction error, the initial model is iteratively trained end-to-end using the training dataset to obtain the object detection model; wherein, the prediction error is calculated based on the classification loss and the bounding box regression loss.

7. The pedestrian detection method based on severe weather as described in claim 2, characterized in that, The speckle pattern is generated based on the Hadamard matrix.

8. A pedestrian detection system based on severe weather, characterized in that, include: The acquisition module is used to acquire the modulated light field sequence and the corresponding light intensity signal sequence; The modulated light field sequence is generated based on multiple sequentially arranged speckle patterns, the modulated light field sequence includes multiple modulation patterns, and the light intensity signal sequence is time-synchronized with the modulated light field sequence; The encoding and calculation module is used to input the modulation pattern of the modulation light field sequence and the light intensity signal sequence into the target detection model, and to perform feature encoding and correlation calculation on the two sequences through the backbone network of the target detection model to obtain a correlation feature map that represents the correlation information of the light field. The detection and recognition module is used to obtain pedestrian detection results within the target area based on the target detection model and the associated feature map.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the pedestrian detection method based on severe weather as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the pedestrian detection method based on severe weather as described in any one of claims 1 to 7.

Images