Pedestrian anomaly detection method, device and storage medium

Abstract

The application relates to a pedestrian anomaly detection method and device and a storage medium, wherein the method comprises the following steps: inputting a first modality image and a second modality image into a target detection model to respectively extract a first modality feature map and a second modality feature map; calculating the difference between the first modality feature map and the second modality feature map via a differential attention mechanism module in the target detection model to generate a feature difference map, and calculating an attention weight based on the feature difference map; fusing the first modality feature map and the second modality feature map based on the attention weight via a feature fusion module in the target detection model to obtain a fused feature map; inputting the fused feature map into a detection head in the target detection model, and outputting a predicted pedestrian position and a predicted pedestrian staying state; and generating a pedestrian anomaly detection result based on the predicted pedestrian position and the predicted pedestrian staying state. Through the application, the problem of low accuracy of pedestrian anomaly detection is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular, to a pedestrian anomaly detection method, device and storage medium. BACKGROUND

[0002] Under the background of smart city construction, pedestrian anomaly detection technology has become one of the key technologies to ensure public safety, optimize resource allocation, and improve service quality. In crowded public places such as shopping malls, stations, schools, etc., timely detection of abnormal staying behavior of pedestrians can effectively prevent safety accidents such as stampedes; in the field of transportation, accurate identification of the staying state of pedestrians in areas such as intersections and zebra crossings can help optimize traffic signal control and reduce traffic accidents; in the business scenario, analyzing the staying behavior of pedestrians in front of shops and commodity display areas can provide important basis for precise marketing and shop layout optimization for businesses.

[0003] In related technologies, pedestrian abnormal staying behavior is detected based on video images. This kind of method acquires two-dimensional image information through a camera, identifies pedestrians using a target detection algorithm, and analyzes the motion state of the pedestrians to determine whether they are staying. However, this method has many limitations in complex environments. In low-light environments (such as at night, in underground parking lots), the contrast and clarity of the image are greatly reduced, and key information such as the facial features and clothing texture of the pedestrians become blurred, making it difficult for the target detection algorithm to accurately extract the features of the pedestrians, resulting in missed detection. In strong light, back light, and other situations, the image is prone to overexposure, shadows, and other problems, further reducing the accuracy of detection. In addition, when pedestrians are mutually occluded or in complex backgrounds, the detection method based on video images has difficulty in distinguishing pedestrians from the background, which can easily cause false detection. Other single-mode detection methods, such as radar-based detection, can overcome the effects of light and occlusion to some extent, but radar data lacks intuitive visual information, making it difficult to accurately classify and analyze the behavior of pedestrians, and it is difficult to meet the needs of fine detection of pedestrian features in actual applications.

[0004] At present, there is no effective solution to the problem of low accuracy of pedestrian anomaly detection in related technologies. SUMMARY

[0005] Embodiments of the present application provide a pedestrian anomaly detection method, device and storage medium to at least solve the problem of low accuracy of pedestrian anomaly detection in related technologies.

[0006] In a first aspect, the embodiments of the present application provide a pedestrian anomaly detection method,

[0007] In some embodiments, the method further comprises:

[0008] extracting pedestrian motion optical flow information of consecutive frames based on the first modality image;

[0009] inputting the first modality image, the second modality image and the pedestrian motion optical flow information into the target detection model to respectively extract features to obtain the first modality feature map, the second modality feature map and the spatiotemporal modality feature map;

[0010] calculating differences between the first modality feature map, the second modality feature map and the spatiotemporal modality feature map via the differential attention mechanism module to generate the feature difference map, and calculating the attention weight based on the feature difference map;

[0011] fusing the first modality feature map, the second modality feature map and the spatiotemporal modality feature map via the feature fusion module based on the attention weight to obtain the fused feature map.

[0012] In some embodiments, the calculating differences between the first modality feature map, the second modality feature map and the spatiotemporal modality feature map via the differential attention mechanism module to generate the feature difference map includes:

[0013] calculating an absolute difference between the first modality feature map and the second modality feature map via the differential attention mechanism module to obtain a first difference map, calculating a relative difference between the first modality feature map and the spatiotemporal modality feature map to obtain a second difference map, and calculating a structural difference between the second modality feature map and the spatiotemporal modality feature map to obtain a third difference map;

[0014] performing weighted average fusion on the first difference map, the second difference map and the third difference map to obtain the feature difference map.

[0015] In some embodiments, the calculating the attention weight based on the feature difference map includes:

[0016] obtaining historical pedestrian trajectory data under the target scene;

[0017] inputting the historical pedestrian trajectory data into the target detection model, and calculating the attention weight according to the historical pedestrian trajectory data and the feature difference map by using the differential attention mechanism module.

[0018] In some embodiments, the calculating the attention weight according to the historical pedestrian trajectory data and the feature difference map by using the differential attention mechanism module includes:

[0019] converting the historical pedestrian trajectory data into a trajectory feature map;

[0020] Based on the mapping relationship between the average position of pedestrians and the trajectory feature map in the historical pedestrian trajectory data, the trajectory feature weights are calculated;

[0021] The attention weight is calculated based on the trajectory feature weights, the trajectory feature map, and the feature difference map.

[0022] In some embodiments, the step of fusing the first modality feature map and the second modality feature map based on the attention weights via the feature fusion module in the target detection model to obtain a fused feature map includes:

[0023] The first modal image is detected, and a scene complexity assessment result is generated based on the detected image indicators; the image indicators include the number of targets and the degree of background change.

[0024] The attention weights are adjusted based on the scene complexity assessment results to obtain new feature weights;

[0025] The first modality feature map and the second modality feature map are fused based on the new feature weights via the feature fusion module to obtain the fused feature map.

[0026] In some embodiments, generating pedestrian anomaly detection results based on the predicted pedestrian location and the predicted pedestrian dwelling state includes:

[0027] Based on the predicted pedestrian location and the predicted pedestrian dwelling state, the abnormal dwelling time is calculated;

[0028] The abnormal dwell time is compared with a preset dwell time threshold, and the pedestrian anomaly detection result is generated based on the comparison result.

[0029] In some embodiments, the first modal image is a visual image, and / or the second modal image is an infrared image.

[0030] Secondly, embodiments of this application provide a pedestrian anomaly detection device, the device comprising:

[0031] The acquisition module is used to acquire a first modal image and a second modal image collected for the same target scene; and to extract pedestrian motion optical flow information of consecutive frames based on the first modal image.

[0032] The feature extraction module is used to input the first modal image, the second modal image, and the pedestrian motion optical flow information into the trained target detection model, and extract features to obtain the first modal feature map, the second modal feature map, and the spatiotemporal modal feature map, respectively;

[0033] The differential attention module is used to calculate the differences between the first modality feature map, the second modality feature map and the spatiotemporal modality feature map via the differential attention mechanism module in the target detection model, generate multiple sets of feature difference maps, and calculate attention weights based on the feature difference maps;

[0034] The feature fusion module is used to fuse the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map based on the attention weights, via the feature fusion module in the target detection model, to obtain a fused feature map; the fused feature map is input to the detection head in the target detection model, and the predicted pedestrian dwell state is output;

[0035] The generation module is used to generate pedestrian anomaly detection results based on the predicted pedestrian dwelling state.

[0036] Thirdly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the pedestrian anomaly detection method as described in the first aspect above.

[0037] Compared to related technologies, the pedestrian anomaly detection method, apparatus, and storage medium provided in this application acquire a first modal image and a second modal image collected for the same target scene; input the first modal image and the second modal image into a trained target detection model to extract features to obtain a first modal feature map and a second modal feature map respectively; calculate the difference between the first modal feature map and the second modal feature map through the differential attention mechanism module in the target detection model to generate a feature difference map, and calculate attention weights based on the feature difference map; fuse the first modal feature map and the second modal feature map based on the attention weights through the feature fusion module in the target detection model to obtain a fused feature map; input the fused feature map into the detection head in the target detection model and output the predicted pedestrian position and predicted pedestrian dwell state; and generate a pedestrian anomaly detection result based on the predicted pedestrian position and predicted pedestrian dwell state.

[0038] By fully integrating the advantages of multimodal data such as image recognition and attention mechanisms, this approach deeply optimizes pedestrian lingering detection tasks, effectively capturing the dynamic changes of pedestrians in different scenarios. Simultaneously, it utilizes differential attention mechanisms to dynamically analyze and weight the feature differences between different modalities, focusing on the details of pedestrian target features and strengthening the capture of pedestrian motion state features. The processed features are then fused, enabling the model to more accurately identify pedestrian lingering states. This achieves efficient and accurate pedestrian lingering detection, effectively solving the problem of low accuracy in pedestrian anomaly detection.

[0039] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0041] Figure 1 This is a hardware structure block diagram of a terminal for a pedestrian anomaly detection method according to an embodiment of this application;

[0042] Figure 2 This is a flowchart of a pedestrian anomaly detection method according to an embodiment of this application;

[0043] Figure 3 This is a system block diagram of a pedestrian anomaly detection method according to an embodiment of this application;

[0044] Figure 4 This is a flowchart of another pedestrian anomaly detection method according to an embodiment of this application;

[0045] Figure 5 This is a structural block diagram of a pedestrian anomaly detection device according to an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0047] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of a terminal for a pedestrian anomaly detection method according to an embodiment of this application. Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0050] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the pedestrian anomaly detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0052] This embodiment provides a method for detecting pedestrian anomalies. Figure 2 This is a flowchart of a pedestrian anomaly detection method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0053] Step S210: Acquire the first modal image and the second modal image for the same target scene.

[0054] This process involves simultaneously acquiring image data in two different modalities (such as RGB images and infrared thermal imaging, visible light and depth maps) for the same target scene using multimodal sensors or imaging devices. These two modalities can cover complementary information about the scene (such as texture details and thermal radiation features, spatial structure and distance information) to ensure that subsequent analysis can comprehensively utilize the key features of multi-source data. During acquisition, spatial alignment of the images must be ensured (e.g., through calibration or registration) to avoid affecting subsequent processing due to differences in viewpoint or scale.

[0055] In an optional embodiment, the first modal image is a visual image, and / or the second modal image is an infrared image. Specifically, image data of the target scene is acquired simultaneously by a visual camera and an infrared camera to obtain visual and infrared images in the same time and space. It should also be understood that, in this embodiment, images with a greater number of modalities can also be used.

[0056] Step S220: Input the first modality image and the second modality image into the trained target detection model, and extract features to obtain the first modality feature map and the second modality feature map respectively.

[0057] Specifically, the acquired first and second modal images are preprocessed, including scaling the images to the size suitable for the target detection model, normalizing pixel values, and removing noise using median filtering; for infrared image thermal noise, bilateral filtering is used to optimize image quality.

[0058] The preprocessed multimodal images are then input into a pre-trained object detection model, such as the object detection algorithm YOLO (You Only Look Once) or a faster multi-branch architecture of the region-based convolutional neural network (Faster R-CNN). This object detection model processes different types of images through independent convolutional backbone networks (such as ResNet-50); for example, for visual images, multi-layer convolutional networks are used to extract visual features of pedestrian appearance and color, while for infrared images, pedestrian thermal radiation features are extracted, generating corresponding first-modality feature maps and second-modality feature maps respectively. These feature maps are typically three-dimensional tensors (height × width × number of channels), preserving semantic information at different levels (such as low-level edges, mid-level textures, and high-level object category features), providing a foundation for subsequent difference analysis and fusion.

[0059] In step S230, the difference between the first modality feature map and the second modality feature map is calculated via the differential attention mechanism module in the object detection model to generate a feature difference map, and attention weights are calculated based on the feature difference map.

[0060] In this step, the feature maps of the different modalities mentioned above are input into the differential attention mechanism module in the object detection model. This module first calculates the element-wise difference (e.g., absolute difference or squared difference) between the two feature maps, generating an initial feature difference map to highlight the complementary information of the two modalities (e.g., the difference between RGB texture details and infrared thermal radiation). Subsequently, the difference map is feature-adjusted through convolutional layers or attention modules (e.g., channel attention, spatial attention) to generate an attention weight map. This weight map is used to quantify the significance of differences at different spatial locations or channels, guiding subsequent fusion to focus on key difference regions (e.g., target edges or modality-specific features). Thus, higher weights can be assigned to key parts of pedestrian targets and areas where motion states change abruptly, strengthening the capture of features related to abnormal stops; while weights are reduced for background and irrelevant areas to suppress interfering information.

[0061] Therefore, the differential attention mechanism generates attention weights by dynamically calculating the feature difference map between modalities, which can effectively highlight key information such as pedestrian edges and thermal radiation differences, suppress redundant background features, and improve feature discrimination. This avoids the contradiction between information redundancy and computational efficiency in multimodal detection, and provides an efficient solution for pedestrian anomaly detection in complex scenarios. It realizes the design of the differential attention mechanism and the adaptive fusion of cross-modal features.

[0062] Step S240: The first modality feature map and the second modality feature map are fused based on attention weights via the feature fusion module in the target detection model to obtain a fused feature map; the fused feature map is input into the detection head in the target detection model and the predicted pedestrian position and predicted pedestrian dwell state are output.

[0063] The feature fusion module uses a weighted map generated by differential attention to perform weighted fusion of the first and second modal feature maps. Specifically, the attention weight map serves as the fusion coefficient, linearly combining corresponding positions or channels of the two modal feature maps to generate a fused feature map. This fused feature map integrates complementary information from the two modalities and suppresses redundant or noisy features (such as background interference). Subsequently, the fused feature map is input into the detection head of the object detection model (such as a fully connected layer or convolutional layer containing classification and regression branches), outputting the predicted pedestrian location (such as bounding box coordinates) and the predicted pedestrian dwell state (such as a binary classification result of stationary or moving).

[0064] It should also be noted that, in this embodiment, taking the improved YOLOv8 model architecture of the aforementioned object detection model as an example, its feature extraction network and detection head are deeply modified, and the differential attention mechanism module is strengthened to make it more efficient in fusing multimodal features. The training process of this object detection model is described in detail below:

[0065] Training was performed using the PyTorch framework (an open-source deep learning framework based on Python), with an initial learning rate of 0.001. The Adam optimizer with weight decay (AdamW) was used, combined with a cosine annealing learning rate adjustment strategy. The training epochs were set to 300, with a batch size of 40. The loss function, based on the original object detection loss and class classification loss in YOLOv8, incorporated a combination of weighted focus loss and dice loss, specifically targeting pedestrian abnormal lingering detection to optimize learning performance on small targets and difficult samples. During training, TensorBoard was used to monitor the loss values, mean average precision (mAP), precision, and recall on both the training and validation sets in real time. Model performance was monitored in real time using the validation set, and training continued if the validation set mAP did not improve for 25 consecutive epochs. If it improved, training continued. An early stopping strategy was triggered when the mean precision on the validation set remained unchanged for 25 consecutive epochs, saving the optimal model parameters. The model is then optimized by using model pruning and quantization techniques to remove redundant structures and reduce parameter precision, and then deployed to edge computing devices.

[0066] Step S250: Based on the predicted pedestrian location and predicted pedestrian dwelling state, generate pedestrian anomaly detection results.

[0067] Based on the predicted pedestrian location and dwelling state output by the detection head, pedestrian anomaly detection results can be generated. More specifically, by combining spatial location information and dwelling state, it is determined whether a pedestrian is in an abnormal scenario (such as remaining stationary in a prohibited area for an extended period of time, or moving in a dangerous area). For example, if the model predicts that a pedestrian has stayed in an unspecified area for more than a preset time threshold, or that their location conflicts with the semantics of the scene (such as a road or restricted area), it is determined to be an abnormal event. The final results are output in the form of alarm signals, visual annotations, or structured data for use by subsequent monitoring or decision-making systems.

[0068] Among related technologies, methods for detecting abnormal pedestrian behavior based on video images are prone to missed detections, and their accuracy is low under conditions such as direct sunlight or backlighting. Detection methods using sensors such as radar lack intuitive visual information, which also affects their accuracy.

[0069] In addition, some multimodal detection methods attempt to fuse multiple data sources such as video, depth, and infrared data to improve detection performance. However, most current multimodal fusion methods suffer from the following problems: First, they employ simple data concatenation or weighted fusion strategies, such as directly concatenating the feature vectors of video images with those of depth images. This approach does not fully consider the inherent connections and complementary characteristics between different modal data, resulting in redundant information in the fused features and failure to effectively enhance key features. Second, they lack optimization mechanisms for pedestrian lingering detection tasks, failing to deeply analyze the characteristics of pedestrian lingering behavior and effectively extract information that plays a crucial role in lingering detection, thus limiting the improvement in detection accuracy and robustness. Third, they perform poorly in terms of real-time performance. Complex multimodal data processing and fusion often require significant computational resources and time, making it difficult to meet the requirements for real-time detection of pedestrian lingering states in real-world scenarios.

[0070] In comparison, the embodiments of this application, through steps S210 to S250, fully integrate the advantages of multimodal data such as image recognition and attention mechanisms, and deeply optimize the pedestrian lingering detection task, effectively capturing the dynamic changes of pedestrians in different scenarios. Simultaneously, by utilizing a differential attention mechanism to dynamically analyze and weight the feature differences of different modal data, focusing on the details of pedestrian target features, and strengthening the capture of pedestrian motion state features, the processed features are then fused, enabling the model to more accurately identify the pedestrian's lingering state. This achieves efficient and accurate pedestrian lingering detection, effectively solving the problem of low accuracy in pedestrian anomaly detection.

[0071] In some embodiments, the above-described pedestrian anomaly detection method may further include the following steps:

[0072] Based on the first modality image, optical flow information of pedestrian motion is extracted from consecutive frames. Specifically, optical flow is used to process the consecutive frame images to extract the optical flow information of pedestrian motion, which is then incorporated into the spatiotemporal features. Optical flow, as a representation of spatiotemporal continuity, can effectively supplement the instantaneous features of a single static image. By tracking the motion trajectory of key points of the pedestrian (such as the head and joints), walking speed, direction, and limb movement patterns can be accurately quantified. This dynamic information, along with position and dwell state, forms a spatiotemporal joint feature, significantly improving detection capabilities in complex scenes.

[0073] Next, the first modal image, the second modal image, and the pedestrian motion optical flow information are input into the target detection model, and feature extraction is performed to obtain the first modal feature map, the second modal feature map, and the spatiotemporal modal feature map. Similar to the pedestrian anomaly detection method in the above embodiment, in this embodiment, feature extraction is performed on each modal image through the feature extraction layer of the target detection model to obtain various feature maps.

[0074] The differential attention mechanism module calculates the differences between the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map to generate a feature difference map. Based on this feature difference map, attention weights are calculated. Specifically, this module first performs pairwise difference calculations (such as absolute difference or squared difference) on the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map to generate three sets of initial feature difference maps, representing complementary information of visible light-infrared, visible light-spatiotemporal, and infrared-spatiotemporal modalities (such as differences between texture and thermal radiation, and comparisons between dynamic trajectories and static features). Subsequently, the difference maps are adjusted for channel dimensions and nonlinear enhancement through convolutional layers and activation functions (such as ReLU) to generate a dynamic attention weight map. This weight map quantifies the significance of differences between different spatial locations and channels, enabling the model to automatically focus on key modal interaction regions (such as modality-specific features or abnormal motion patterns at pedestrian edges) while suppressing redundant background interference, providing learnable weight guidance for subsequent feature fusion.

[0075] Subsequently, based on attention weights, the feature fusion module fuses the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map to obtain a fused feature map. Specifically, the module multiplies the first modality, the second modality, and the spatiotemporal modality feature maps element-wise with their corresponding attention weight maps, weighting and strengthening significantly different feature channels or spatial regions, and suppressing redundant features in the common modality. For example, in a motion scene, the dynamic trajectory features of the spatiotemporal modality may be given higher weights, while the common modality features of the static background are weakened.

[0076] The fused feature map integrates complementary information and dynamic interaction characteristics from multiple modalities. It retains the texture details of visible light and the thermal radiation features of infrared light, while incorporating spatiotemporal continuity information (such as optical flow trajectories). Finally, it is input into the detection head for pedestrian position regression and dwell state classification. This achieves improved environmental adaptability by combining complementary information from visible light (texture details), infrared (thermal radiation features), and spatiotemporal modalities (optical flow trajectories). In complex scenarios such as sudden changes in lighting (e.g., at night) and occlusion (e.g., in dense crowds), other modalities can still provide effective features when a single modality fails. For example, the infrared modality can replace the visible light modality to maintain detection capability at night, and the spatiotemporal modality can penetrate temporary occlusions to continuously track pedestrians through optical flow trajectories. Furthermore, the spatiotemporal joint judgment effectively improves the accuracy of anomaly detection.

[0077] In some embodiments, the above-mentioned calculation of the differences between the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map via the differential attention mechanism module to generate a feature difference map may further include the following steps:

[0078] The differential attention mechanism module calculates the absolute difference between the first modal feature map and the second modal feature map to obtain the first difference map. It then calculates the relative difference between the first modal feature map and the spatiotemporal modal feature map to obtain the second difference map. Finally, it calculates the structural difference between the second modal feature map and the spatiotemporal modal feature map to obtain the third difference map. The first difference map, the second difference map, and the third difference map are then fused by weighted average to obtain the feature difference map.

[0079] Specifically, the feature maps are first preprocessed. The three modal feature maps are uniformly adjusted to the same number of channels by 1X1 convolution, and are denoted as visual modal feature map V, infrared modal feature map I and spatiotemporal modal feature map F. The size of each feature map is [B,C,H,W], where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map.

[0080] A multi-strategy fusion approach is used to calculate the differences between the three sets of modal feature maps, generating a feature difference map. For example, the feature difference map between the visual modal feature map and the infrared modal feature map is calculated using absolute differences, the visual modal feature map and the spatiotemporal modal feature map is calculated using relative differences, and the infrared modal feature map and the spatiotemporal modal feature map are calculated using structural differences. The calculation method is shown in the following formula:

[0081] D{m1,m2}= Conv1x1(M(m1,m2));

[0082] Where m1 and m2 represent two different modal feature maps, such as visual modal feature map V and infrared modal feature map I, M(·) is the difference calculation function. The generated visual-infrared difference map D{V,I} (i.e., the first difference map mentioned above), visual-spatiotemporal difference map D{V,F} (i.e., the second difference map mentioned above), and infrared-spatiotemporal difference map D{I,F} (i.e., the third difference map mentioned above) are weighted and averaged to obtain the final cross-modal feature difference map D{fused}.

[0083] Through the above embodiments, the weighted average fusion of three sets of difference maps not only preserves the specific information of each modality (such as the robustness of thermal radiation to occlusion), but also suppresses redundant features through weight allocation, thus achieving a balance between detection accuracy and computational efficiency.

[0084] In some embodiments, the above-described calculation of attention weights based on feature difference maps may further include the following steps:

[0085] Acquire historical pedestrian trajectory data in the target scene; input the historical pedestrian trajectory data into the target detection model, and use the differential attention mechanism module to calculate the attention weight based on the historical pedestrian trajectory data and feature difference map.

[0086] In this step, the attention allocation mechanism is optimized by integrating spatiotemporal context information. Specifically, historical pedestrian trajectory data (such as location sequence, movement speed, and dwelling hotspots) in the target scene is first acquired and input into the target detection model as prior knowledge. Subsequently, the differential attention mechanism module combines historical trajectory data with multimodal feature difference maps (such as visible light-infrared differences and spatiotemporal dynamic differences) to generate a spatiotemporally perceptive attention weight map by analyzing the temporal patterns of pedestrian behavior. This weight map not only reflects the modal difference significance of the current frame (such as the thermal radiation contrast of pedestrian edges) but also incorporates the statistical features of historical trajectories (such as resident areas and movement tendencies). This allows the model to dynamically focus on areas strongly correlated with historical behavior patterns (such as abnormal movements that suddenly deviate from the normal path) during feature fusion, while suppressing irrelevant background interference (such as the comodal features of static vegetation).

[0087] In some embodiments, the differential attention mechanism module described above, which calculates attention weights based on historical pedestrian trajectory data and feature difference maps, may further include the following steps:

[0088] Historical pedestrian trajectory data is transformed into trajectory feature maps; based on the mapping relationship between the average position of pedestrians and the trajectory feature maps in the historical pedestrian trajectory data, trajectory feature weights are calculated; based on the trajectory feature weights, trajectory feature maps, and feature difference maps, attention weights are calculated.

[0089] The process of assigning attention weights based on historical movement trajectories and feature difference maps is as follows: First, the historical trajectories are encoded, transforming the pedestrian historical movement trajectory data into a trajectory feature map T that can be fused with the feature map. Then, the average position of the pedestrian in the trajectory is calculated and mapped to the corresponding region of the feature map, and trajectory feature weights are generated based on the proportion of dwell time.

[0090] Then, convolution is used to extract the spatiotemporal features of the trajectory, forming a trajectory feature representation that matches the size of the modality feature map. An improved differential attention mechanism is introduced to dynamically assign weights by combining the feature difference map and the trajectory features, as shown in the following formula:

[0091] ;

[0092] In the above formula, A is the final attention weight map, A S For spatial attention weights, A C For channel attention weights, This represents element-wise multiplication. Spatial attention A S By concatenating the aforementioned fused feature difference map D{fused} with the trajectory feature map based on trajectory feature weights, and then calculating it through a convolutional layer and activation function, the focus is on key parts of the pedestrian and regions of sudden motion changes; channel attention A CBy performing global average pooling on the multimodal fusion feature map, and generating it through a multilayer perceptron (MLP) and activation function, the feature channels that contribute to dwell detection are highlighted.

[0093] Through the above embodiments, by encoding historical pedestrian trajectory data into trajectory feature maps and generating trajectory feature weights based on the average pedestrian position mapping and dwell time, and finally dynamically calculating attention weights by combining feature difference maps, the model's ability to perceive the spatiotemporal context of abnormal behavior is significantly enhanced. By guiding the attention mechanism to accurately focus on high-risk targets through continuous behavioral patterns (such as long-term lingering areas) provided by historical trajectories, false alarms caused by instantaneous environmental interference are effectively suppressed. At the same time, the detection sensitivity of disguised anomalies (such as slowly moving loiterers) is improved, significantly improving the accuracy of pedestrian anomaly detection in complex scenarios and helping to reduce the target loss rate.

[0094] In some embodiments, the above-mentioned feature fusion module in the object detection model fuses the first modality feature map and the second modality feature map based on attention weights to obtain a fused feature map, and may further include the following steps:

[0095] The first modality image is detected, and a scene complexity assessment result is generated based on the detected image indicators. The image indicators include the number of targets and the degree of background change. The attention weights are adjusted according to the scene complexity assessment result to obtain new feature weights. The first modality feature map and the second modality feature map are fused based on the new feature weights through the feature fusion module to obtain a fused feature map.

[0096] In this step, dynamic calibration of attention weights is primarily achieved through scene complexity assessment. Specifically, the number of targets (e.g., pedestrian density) and the degree of background change (e.g., sudden changes in illumination, interference from dynamic objects) in the first modality image are detected, generating a scene complexity assessment result. High-complexity scenes (dense targets, dynamic background) trigger a weight adjustment mechanism, reducing attention allocation to susceptible modalities (e.g., overexposure features of the visible light modality under strong light) and instead increasing the weight of more robust modalities (e.g., thermal radiation features of the infrared modality or optical flow trajectories of the spatiotemporal modality). Low-complexity scenes maintain a balanced weight allocation to preserve multimodal complementarity. For example, in a densely populated concert scene, the model suppresses background noise in the visible light modality and focuses on the thermal radiation features of the infrared modality and the group motion trajectories of the spatiotemporal modality, thereby improving detection stability.

[0097] The feature fusion module achieves scene-adaptive fusion of multimodal features based on adjusted new feature weights. Specifically, the module multiplies the feature maps of the first modality (e.g., vision) and the second modality (e.g., infrared) element-wise with a dynamically generated weight map. This weights and strengthens the feature channels or spatial regions of key modalities (e.g., infrared) in complex scenes, while suppressing redundant features of non-key modalities. For example, in a rapidly changing traffic intersection scene, the new weights assign higher weights to the optical flow trajectory of the spatiotemporal modality, making the fused feature map more focused on the dynamic interaction between vehicles and pedestrians, while weakening the noise features in the visible light modality caused by illumination changes. The final fused feature map input to the detection head combines multimodal complementarity with scene adaptability, improving detection accuracy.

[0098] Through the above embodiments, dynamic adjustment of attention weights driven by scene complexity achieves adaptive optimization of multimodal detection, making the fused feature map more focused on key information and improving the accuracy of pedestrian anomaly detection. Furthermore, by automatically calibrating weights based on target density and background dynamics, more robust modal features are prioritized in complex scenes (such as dense crowds or sudden changes in lighting), effectively reducing the false alarm rate (measured reduction of 25%-30%). In addition, the above weight adjustment mechanism does not require model retraining and can quickly adapt to different scenes (such as switching from an indoor corridor to an outdoor plaza), reducing deployment costs.

[0099] In some embodiments, the process of generating pedestrian anomaly detection results based on predicted pedestrian location and predicted pedestrian dwelling state may further include the following steps:

[0100] Based on the predicted pedestrian location and predicted pedestrian dwelling state, the abnormal dwelling time is calculated; the abnormal dwelling time is compared with the preset dwelling time threshold, and the pedestrian anomaly detection result is generated based on the comparison result.

[0101] Specifically, based on the predicted pedestrian location (spatial coordinates) and dwelling state (e.g., stationary, slightly moving), time-series tracking algorithms (e.g., Kalman filtering) are used to statistically analyze the continuous dwelling time of pedestrians within the same spatial area, generating the original dwelling time. In another embodiment, the original time can be calibrated by combining pedestrian historical trajectory data (e.g., frequented areas, movement habits) to eliminate misjudgments caused by short waits (e.g., waiting at red lights, waiting for a bus), ultimately obtaining an accurate abnormal dwelling time—this time reflects both the actual physical duration of the pedestrian's stay and incorporates contextual information from historical behavioral patterns (e.g., whether they deviated from their usual path). For example, in a station scenario, the system excludes the normal waiting time of passengers on seats and only records abnormal dwelling times for pedestrians who suddenly remain outside the waiting area.

[0102] Subsequently, based on the security level of the target scenario (such as different requirements for shopping malls and subway stations), a preset dwell time threshold (e.g., 30 seconds) is established. The calculated abnormal dwell time is then compared with the threshold in real time. If the threshold is exceeded, a secondary verification mechanism is triggered (e.g., combining optical flow tracing to determine whether the person is stationary or moving slowly). Once the anomaly is confirmed, a detection result (e.g., "illegal stay" or "fainting warning") is generated. If the threshold is not exceeded, the behavior is considered normal. Furthermore, the threshold supports dynamic adjustment, automatically widening or tightening according to the complexity of the scenario (e.g., a surge in pedestrian traffic during holidays) to balance the risks of false alarms and missed alarms.

[0103] The present application will now be described in detail with reference to specific embodiments. Figure 3 This is a system block diagram of a pedestrian anomaly detection method according to an embodiment of this application, such as... Figure 3 As shown, the system includes: a data acquisition layer, a feature processing layer, a YOLO detection head module, an anomaly detection module, and a result output module; wherein:

[0104] The data acquisition layer is used to input the visual images captured by the visual camera, the infrared images captured by the infrared camera, and the pedestrian motion optical flow information calculated by the optical flow calculation module into the multimodal data acquisition module.

[0105] The feature processing layer includes a data preprocessing module, a modal feature extraction module, a differential attention mechanism module, a feature fusion module, and a feature optimization module.

[0106] Specifically, the data preprocessing module preprocesses the images input from the multimodal data acquisition module, and then inputs the preprocessed images of each modality into the corresponding visual feature extraction module, infrared feature extraction module, and spatiotemporal feature extraction module: visual images are processed through a multi-layer convolutional neural network to extract visual features such as pedestrian appearance and color; infrared images are used to extract pedestrian thermal radiation features; and optical flow information is processed through a specially designed spatiotemporal convolution module to extract the spatiotemporal features of pedestrian motion. This sequentially yields visual modal feature maps, infrared modal feature maps, and spatiotemporal modal feature maps.

[0107] The differential attention mechanism module performs in-depth processing. An improved differential attention mechanism is introduced, first calculating the pairwise differences between the visual modality feature map, infrared modality feature map, and spatiotemporal modality feature map, generating multiple sets of feature difference maps. Then, through the attention calculation module, combined with the pedestrian's historical motion trajectory data (if available), weights are dynamically assigned to features of different modalities and regions. Higher weights are given to key parts of the pedestrian target and areas where the motion state changes abruptly, strengthening the capture of features related to abnormal stops; for background and irrelevant areas, the weights are reduced to suppress interfering information.

[0108] The feature fusion module and feature optimization module then perform corresponding processing. The three modal feature maps processed by the differential attention mechanism are fused. An adaptive weighted fusion strategy is adopted, dynamically adjusting the weights of each modal feature fusion based on the weights obtained from the differential attention mechanism and the scene complexity evaluation results (such as by calculating the number of targets in the image, the degree of background change, etc.) to generate the fused feature map. The fused feature map is further optimized by using residual connections and a feature pyramid structure to enhance the expressive power and scale adaptability of the features.

[0109] The YOLO detection head module takes the fused and optimized feature map from the input and performs convolution, pooling, and fully connected operations to predict the location, category, and dwelling state of pedestrian targets. The anomaly detection module introduces abnormal behavior judgment rules, combining pedestrian dwelling time thresholds (e.g., continuous dwelling exceeding 30 seconds is considered abnormal) and dwelling area attributes (e.g., prohibited dwelling areas) to comprehensively determine whether a pedestrian is in an abnormal dwelling state. Finally, the result output module outputs the pedestrian abnormal dwelling detection results.

[0110] Taking the YOLO-based multimodal feature differential attention fusion pedestrian dwell detection method as an example, Figure 4 This is a flowchart of another pedestrian anomaly detection method according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps:

[0111] Step S401: In key areas such as the waiting hall, entrance passage, exit passage, and security checkpoint of a large railway station, install high-definition visual cameras (resolution 3840×2160) and infrared cameras (thermal imaging resolution 1024×768). The cameras are installed above and on both sides at a height of 5 meters. Through reasonable layout, ensure that the field of view of the two types of cameras overlaps at all heights to provide comprehensive coverage of key monitoring areas.

[0112] In step S402, image data is continuously collected for 15 days at a frequency of 25 frames per second during the entire operating period of the train station (from 5 a.m. to 1 a.m. the next day), covering weekdays and holidays, and a total of approximately 32,400,000 visual-infrared image pairs are obtained.

[0113] Step S403 involves preprocessing the data using Python's OpenCV, imgaug, and NumPy libraries. Images are uniformly scaled to 1280×1280 pixels, and pixel values ​​are normalized to the 0-1 range. Non-local mean filtering is used to remove visual image noise, and median filtering and adaptive histogram equalization are combined to enhance image contrast and clarity. The DeepFlow optical flow algorithm is used to calculate optical flow information between consecutive frames to obtain the spatiotemporal characteristics of pedestrian motion. The imgaug library is then used to perform data augmentation operations such as random flipping, rotation, and brightness adjustment on the visual images, expanding the diversity of the dataset.

[0114] Step S404: Divide the preprocessed dataset into training set, validation set and test set in a ratio of 7:1.5:1.5.

[0115] Step S405: Based on the YOLOv8 model architecture, embed the differential attention mechanism module, deeply modify its feature extraction network and detection head, strengthen the differential attention mechanism module, and make it more efficient in integrating multimodal features.

[0116] Step S406: The model is trained using the PyTorch framework with an initial learning rate of 0.001. The AdamW optimization algorithm is used, combined with a cosine annealing learning rate adjustment strategy. The training epochs are set to 300, and the batch size is 40. The loss function, based on the original object detection loss and category classification loss in YOLOv8, introduces a combination of weighted Focal Loss and Dice Loss to optimize the learning effect on small targets and difficult samples, specifically for pedestrian abnormal lingering detection tasks. During training, TensorBoard (a visualization tool for monitoring, debugging, and analyzing the training process and performance of machine learning models) is used to monitor the loss values, mean AP, precision, recall, and other metrics on the training and validation sets in real time.

[0117] Step S407: Determine if the validation set mAP has not improved for 25 consecutive rounds. Specifically, during training, the model performance is monitored in real time using the validation set, and it is determined whether the validation set mAO has not improved for 25 consecutive rounds. If there is improvement, model training continues. If not, return to step S406 above.

[0118] Step S408: Early stopping and pruning to optimize the model. This involves triggering an early stopping strategy to save the optimal model parameters. Model pruning and quantization techniques are then applied to remove redundant structures and reduce parameter precision.

[0119] Step S409: Deploy the optimized model to the edge computing device.

[0120] Step S410 involves monitoring the test set and real-time data. Specifically, abnormal pedestrian lingering is detected using the test set and real-time image data collected at the security checkpoint.

[0121] Step S411: Determine whether abnormal loitering behavior has occurred based on the rules. Specifically, based on the actual functions and security requirements of key areas of the train station, the rules for determining abnormal loitering are formulated: pedestrians who stay in the security check queuing area for more than 5 minutes without obvious movement, who stay at the entrance or exit of the security check equipment for more than 30 seconds and obstruct passage, or who loiter in prohibited loitering areas (such as the area with warning signs at the security checkpoint) are all deemed to be loitering abnormally.

[0122] Step S412: Output the pedestrian abnormal stopping detection results; end the process.

[0123] Through the above embodiments, images are simultaneously acquired using visual and infrared cameras, and spatiotemporal information is obtained through optical flow calculation. After preprocessing, the images are input into an improved YOLO model to extract multi-dimensional features. A differential attention mechanism is used to calculate modal feature differences and dynamically allocate weights. Features are then fused using an adaptive weighting strategy based on scene complexity, followed by optimization through residual connections. Finally, based on the prediction results from the detection head, combined with multi-dimensional information such as dwell time thresholds and regional attributes, abnormal pedestrian dwelling is determined. Accordingly, multi-modal data collaborative acquisition and preprocessing improve the system's scene adaptability and data quality; multi-dimensional feature extraction enhances the perception of abnormal features; the differential attention mechanism reduces the false positive and false negative rates; adaptive feature fusion optimization improves model generalization and the ability to detect targets at different scales; and multi-dimensional anomaly determination meets high-precision detection requirements, providing reliable support for practical applications.

[0124] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0125] This embodiment also provides a pedestrian anomaly detection device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as described thereon. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0126] Figure 5 This is a structural block diagram of a pedestrian anomaly detection device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0127] The acquisition module 51 is used to acquire a first modal image and a second modal image collected for the same target scene; and to extract pedestrian motion optical flow information of consecutive frames based on the first modal image.

[0128] The feature extraction module 52 is used to input the first modality image, the second modality image, and the pedestrian motion optical flow information into the trained target detection model, and extract features to obtain the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map, respectively;

[0129] The differential attention module 53 is used to calculate the differences between the first modality feature map, the second modality feature map and the spatiotemporal modality feature map via the differential attention mechanism module in the target detection model, generate multiple sets of feature difference maps, and calculate attention weights based on the feature difference maps;

[0130] The feature fusion module 54 is used to fuse the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map based on attention weights through the feature fusion module in the target detection model to obtain a fused feature map; the fused feature map is input into the detection head in the target detection model and the predicted pedestrian dwell state is output;

[0131] The generation module 55 is used to generate pedestrian anomaly detection results based on the predicted pedestrian dwelling status.

[0132] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0133] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0134] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0135] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0136] S1, acquire the first modal image and the second modal image collected for the same target scene;

[0137] S2, input the first modality image and the second modality image into the trained target detection model, and extract features to obtain the first modality feature map and the second modality feature map respectively;

[0138] S3, via the differential attention mechanism module in the object detection model, calculates the difference between the first modality feature map and the second modality feature map, generates a feature difference map, and calculates the attention weights based on the feature difference map;

[0139] S4, via the feature fusion module in the object detection model, the first modality feature map and the second modality feature map are fused based on attention weights to obtain a fused feature map; the fused feature map is input into the detection head in the object detection model, and the predicted pedestrian position and predicted pedestrian dwell state are output;

[0140] S5 generates pedestrian anomaly detection results based on the predicted pedestrian location and predicted pedestrian dwelling state.

[0141] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0142] Furthermore, in conjunction with the pedestrian anomaly detection methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the pedestrian anomaly detection methods in the above embodiments.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0144] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0145] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting pedestrian anomalies, characterized in that, include: Acquire first and second modal images of the same target scene; The first modality image and the second modality image are input into the trained target detection model, and feature extraction is performed to obtain the first modality feature map and the second modality feature map, respectively. The differential attention mechanism module in the target detection model calculates the difference between the first modality feature map and the second modality feature map to generate a feature difference map, and calculates attention weights based on the feature difference map, including: Obtain historical pedestrian trajectory data in the target scenario; The historical pedestrian trajectory data is input into the target detection model, and the attention weight is calculated using the differential attention mechanism module based on the historical pedestrian trajectory data and the feature difference map. The first modality feature map and the second modality feature map are fused together by the feature fusion module in the target detection model based on the attention weights to obtain a fused feature map; the fused feature map is input into the detection head in the target detection model and the predicted pedestrian position and predicted pedestrian dwell state are output. Based on the predicted pedestrian location and the predicted pedestrian dwelling state, a pedestrian anomaly detection result is generated; The method further includes: Based on the first modal image, extract the pedestrian motion optical flow information of consecutive frames; The first modal image, the second modal image, and the pedestrian motion optical flow information are input into the target detection model, and feature extraction is performed to obtain the first modal feature map, the second modal feature map, and the spatiotemporal modal feature map, respectively. The differential attention mechanism module calculates the differences between the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map to generate the feature difference map, and calculates the attention weight based on the feature difference map; wherein, each set of feature difference maps represents the complementary information of the visible light-infrared, visible light-spatiotemporal, and infrared-spatiotemporal modes, respectively. Based on the attention weights, the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map are fused through the feature fusion module to obtain the fused feature map.

2. The pedestrian anomaly detection method according to claim 1, characterized in that, The step of calculating the differences between the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map via the differential attention mechanism module, and generating the feature difference map, includes: The differential attention mechanism module calculates the absolute difference between the first modal feature map and the second modal feature map to obtain a first difference map, calculates the relative difference between the first modal feature map and the spatiotemporal modal feature map to obtain a second difference map, and calculates the structural difference between the second modal feature map and the spatiotemporal modal feature map to obtain a third difference map. The first difference map, the second difference map, and the third difference map are weighted and averaged to obtain the feature difference map.

3. The pedestrian anomaly detection method according to claim 1, characterized in that, The step of using the differential attention mechanism module to calculate the attention weight based on the historical pedestrian trajectory data and the feature difference map includes: The historical pedestrian trajectory data is converted into a trajectory feature map; Based on the mapping relationship between the average position of pedestrians and the trajectory feature map in the historical pedestrian trajectory data, the trajectory feature weights are calculated; The attention weight is calculated based on the trajectory feature weights, the trajectory feature map, and the feature difference map.

4. The pedestrian anomaly detection method according to claim 1, characterized in that, The feature fusion module in the target detection model fuses the first modality feature map and the second modality feature map based on the attention weights to obtain a fused feature map, including: The first modal image is detected, and a scene complexity assessment result is generated based on the detected image indicators; the image indicators include the number of targets and the degree of background change. The attention weights are adjusted based on the scene complexity assessment results to obtain new feature weights; The first modality feature map and the second modality feature map are fused based on the new feature weights via the feature fusion module to obtain the fused feature map.

5. The pedestrian anomaly detection method according to claim 1, characterized in that, The step of generating pedestrian anomaly detection results based on the predicted pedestrian location and the predicted pedestrian dwelling state includes: Based on the predicted pedestrian location and the predicted pedestrian dwelling state, the abnormal dwelling time is calculated; The abnormal dwell time is compared with a preset dwell time threshold, and the pedestrian anomaly detection result is generated based on the comparison result.

6. The pedestrian anomaly detection method according to any one of claims 1 to 5, characterized in that, The first modal image is a visual image, and / or the second modal image is an infrared image.

7. A pedestrian anomaly detection device, characterized in that, The device includes: The acquisition module is used to acquire a first modal image and a second modal image collected for the same target scene; and to extract pedestrian motion optical flow information of consecutive frames based on the first modal image. The feature extraction module is used to input the first modal image, the second modal image, and the pedestrian motion optical flow information into the trained target detection model, and extract features to obtain the first modal feature map, the second modal feature map, and the spatiotemporal modal feature map, respectively; The differential attention module is used to calculate the differences between the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map via the differential attention mechanism module in the target detection model, generate multiple sets of feature difference maps, and calculate attention weights based on the feature difference maps, including: Obtain historical pedestrian trajectory data in the target scene; input the historical pedestrian trajectory data into the target detection model, and use the differential attention mechanism module to calculate the attention weight based on the historical pedestrian trajectory data and the feature difference map; The feature fusion module is used to fuse the first modality feature map, the second modality feature map, and the spatiotemporal modality feature map based on the attention weights, via the feature fusion module in the target detection model, to obtain a fused feature map; the fused feature map is input to the detection head in the target detection model, and the predicted pedestrian dwell state is output; The generation module is used to generate pedestrian anomaly detection results based on the predicted pedestrian dwelling state.

8. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the pedestrian anomaly detection method according to any one of claims 1 to 6 when it is run.

Images