An abnormal behavior recognition method based on multi-modal perception and related equipment

Abstract

The application discloses an abnormal behavior recognition method based on multi-modal perception and related equipment, and the method comprises the following steps: preprocessing a visible light RGB video frame sequence and an infrared video frame sequence; optimizing a preset first network based on the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence; inputting the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into the optimized first network for feature extraction and fusion processing to obtain fused cross-modal visual features; encoding and aggregating a preset abnormal behavior category text description to generate a text classification weight; calculating the similarity between the fused cross-modal visual features and the text classification weight to obtain an abnormal behavior classification probability, and then determining the abnormal behavior. The application reduces the confusion probability of similar behaviors, improves the accuracy of abnormal behavior discrimination in complex scenes, and can be widely applied to the technical field of behavior recognition.

Description

Technical Field

[0001] This application relates to the field of behavior recognition technology, and in particular to an abnormal behavior recognition method and related equipment based on multimodal perception. Background Technology

[0002] Currently, risk control in border inspection scenarios mainly relies on manual duty and traditional video surveillance. These methods exhibit numerous problems when faced with complex, ever-changing, high-density, and high-frequency passenger flows, including identification lag, large blind spots, and untimely intervention responses. Especially when dealing with typical high-risk behaviors such as climbing over barriers, tailgating, attempting to break through checkpoints, illegal gatherings, and looking around suspiciously, the system often struggles to make timely and accurate judgments. A certain proportion of individuals among those passing through customs exhibit non-intentional risk behaviors, such as those traveling with infants, passengers carrying large amounts of luggage, and children tailgating; under the existing technological architecture, these individuals are highly susceptible to misjudgment or missed detection, thus impacting port management efficiency and security levels.

[0003] However, most existing abnormal behavior recognition solutions rely on images captured by visible light cameras, combined with rule templates, boundary triggers, and area monitoring for abnormal behavior analysis. In complex scenarios such as nighttime, inclement weather, and occlusion, the quality of video images captured by visible light cameras deteriorates, blurring human contours and losing texture details, resulting in low accuracy of behavior recognition at night or in inclement weather. In addition, abnormal behavior recognition solutions usually rely on visual features for behavior classification without incorporating semantic prior information about behavior categories. It is difficult to effectively distinguish between similar behaviors such as "normal queuing" and "abnormal gathering," or "looking left and right" and "normal looking around," leading to confusion between similar behaviors and thus affecting the accuracy of behavior recognition. Summary of the Invention

[0004] The main objective of this application is to propose an abnormal behavior recognition method and related equipment based on multimodal perception, which can reduce the probability of confusion between similar behaviors and improve the accuracy and generalization ability of abnormal behavior judgment in complex scenarios.

[0005] To achieve the above objectives, one aspect of this application proposes an abnormal behavior recognition method based on multimodal perception, including: Acquire the visible light RGB video frame sequence and the infrared video frame sequence of the target to be identified, and preprocess the visible light RGB video frame sequence and the infrared video frame sequence; The preset first network is optimized based on the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain the optimized first network; The preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence are input into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features; The text descriptions of predefined abnormal behavior categories are encoded and aggregated to generate text classification weights; The similarity between the fused cross-modal visual features and the text classification weights is calculated to obtain the abnormal behavior classification probability, and then the abnormal behavior of the target to be identified is determined based on the abnormal behavior classification probability.

[0006] In some embodiments, optimizing a preset first network based on a preprocessed visible light RGB video frame sequence and a preprocessed infrared video frame sequence to obtain an optimized first network includes: The preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence are input into a preset first network for feature extraction to obtain the first visual feature of the visible light RGB video frame sequence and the second visual feature of the infrared video frame sequence. The preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence are input into a preset second network for feature extraction to obtain the third visual feature of the visible light RGB video frame sequence and the fourth visual feature of the infrared video frame sequence. The residual distillation loss is determined based on the first visual feature, the second visual feature, the third visual feature, and the fourth visual feature; The parameters of the preset first network are optimized based on the residual distillation loss to obtain the optimized first network.

[0007] In some embodiments, the step of inputting the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features includes: The optimized first network extracts features from the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain the fifth visual feature of the visible light RGB video frame sequence and the sixth visual feature of the infrared video frame sequence. The fifth visual feature and the sixth visual feature are time-weighted respectively to obtain the first weighted visual feature of the visible light RGB video frame sequence and the second weighted visual feature of the infrared video frame sequence; The first weighted visual feature and the second weighted visual feature are fused by a cross-attention mechanism to obtain a fused cross-modal visual feature.

[0008] In some embodiments, the step of performing temporal weighting on the fifth visual feature and the sixth visual feature respectively to obtain a first weighted visual feature of the visible light RGB video frame sequence and a second weighted visual feature of the infrared video frame sequence includes: The first time weight of each first spatiotemporal feature unit is calculated based on the preset time decay coefficient and the index of each first spatiotemporal feature unit in the fifth visual feature. The second time weight of each second spatiotemporal feature unit is calculated based on the preset time decay coefficient and the index of each second spatiotemporal feature unit in the sixth visual feature. Each of the first spatiotemporal feature units is weighted according to the first time weight to obtain the first weighted visual feature of the visible light RGB video frame sequence. Each of the second spatiotemporal feature units is weighted according to the second time weight to obtain the second weighted visual feature of the infrared video frame sequence.

[0009] In some embodiments, fusing the first weighted visual features and the second weighted visual features through a cross-attention mechanism to obtain fused cross-modal visual features includes: The first weighted visual feature is mapped to a first query matrix, and the second weighted visual feature is mapped to a first key matrix and a first value matrix, respectively. Calculate the similarity between the first query matrix and the first key matrix to obtain the first attention weight matrix, and then normalize the first attention weight matrix. The first attention weight matrix after normalization is summed with the first value matrix to obtain the first cross-modal interaction feature. The second weighted visual feature is mapped to the second query matrix, and the first weighted visual feature is mapped to the second key matrix and the second value matrix, respectively. Calculate the similarity between the second query matrix and the second key matrix to obtain the second attention weight matrix, and then normalize the second attention weight matrix. The second attention weight matrix and the second value matrix are weighted and summed to obtain the second cross-modal interaction feature. Based on the modal balance modulation factor, the first cross-modal interaction feature and the second cross-modal interaction feature are fused to obtain the fused cross-modal visual feature.

[0010] In some embodiments, encoding and aggregating the text descriptions of preset abnormal behavior categories to generate text classification weights includes: The text descriptions of preset behavior categories are segmented to obtain word sequence text. The word text sequence is encoded to obtain a word text feature sequence; Aggregate the word text features in the word text feature sequence to obtain an aggregated feature vector; The aggregated feature vector is normalized to generate text classification weights.

[0011] In some embodiments, calculating the similarity between the fused cross-modal visual features and the text classification weights to obtain the abnormal behavior classification probability includes: Global average pooling is performed on the fused cross-modal visual features to obtain a global feature vector; The global feature vector is multiplied by the text classification weight in the preset abnormal behavior category to obtain the original similarity score of each abnormal behavior category. The original similarity score for each abnormal behavior category is used to calculate the abnormal behavior classification probability through a category mapping function.

[0012] To achieve the above objectives, another aspect of this application proposes an abnormal behavior recognition device based on multimodal perception, the device comprising: The sequence preprocessing module is used to acquire the visible light RGB video frame sequence and the infrared video frame sequence of the target to be identified, and to preprocess the visible light RGB video frame sequence and the infrared video frame sequence. The network optimization module is used to optimize the preset first network based on the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain the optimized first network. The feature fusion module is used to input the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features; The classification weight generation module is used to encode and aggregate the text descriptions of preset abnormal behavior categories to generate text classification weights. An abnormal behavior determination module is used to calculate the similarity between the fused cross-modal visual features and the text classification weights to obtain the abnormal behavior classification probability, and then determine the abnormal behavior of the target to be identified based on the abnormal behavior classification probability.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0015] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides an abnormal behavior recognition method based on multimodal perception. The method first acquires visible light RGB video frame sequences and infrared video frame sequences of the target to be identified, and preprocesses these sequences. Based on the preprocessed visible light RGB video frame sequences and infrared video frame sequences, a preset first network is optimized to obtain an optimized first network. The preprocessed visible light RGB video frame sequences and infrared video frame sequences are input into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features. Preset abnormal behavior category text descriptions are encoded and aggregated to generate text classification weights. The similarity between the fused cross-modal visual features and the text classification weights is calculated to obtain the abnormal behavior classification probability. Then, the abnormal behavior of the target to be identified is determined based on the abnormal behavior classification probability. This application simultaneously introduces visible light and infrared dual-modal information, and effectively overcomes the limitations of single RGB modality in complex scenarios such as low light, backlight, nighttime, and occlusion by fusing the visible light and infrared dual-modal information across modalities. This improves the completeness of pedestrian behavior perception in complex environments and thus enhances the accuracy of abnormal behavior recognition. Furthermore, by using textual semantics as prior information for category semantics and combining it with the fused cross-modal features in the behavior discrimination process, visually similar but semantically different behaviors can be effectively distinguished with the help of prior textual semantic information, reducing the probability of confusion between similar behaviors and improving the accuracy and generalization ability of abnormal behavior discrimination in complex scenarios. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart of an abnormal behavior recognition method based on multimodal perception provided in an embodiment of this application; Figure 3 This is a schematic diagram of infrared video frame images and visible light video frame images provided in the embodiments of this application; Figure 4 This is a schematic diagram of the architecture of the abnormal behavior recognition method based on multimodal perception provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an abnormal behavior recognition device based on multimodal perception provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] It is understood that the terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Most existing abnormal behavior recognition solutions rely on images captured by visible light cameras, combined with rule templates, boundary triggers, and area monitoring for abnormal behavior analysis. However, in complex scenarios such as nighttime, inclement weather, and occlusion, the quality of video images captured by visible light cameras deteriorates, blurring human contours and losing texture details, resulting in low accuracy in behavior recognition at night or in inclement weather. In addition, abnormal behavior recognition solutions typically rely on visual features for behavior classification without incorporating semantic prior information about behavior categories. This makes it difficult to effectively distinguish between similar behaviors such as "normal queuing" and "abnormal gathering," or "looking left and right" and "normal peering," leading to confusion between similar behaviors and thus affecting the accuracy of behavior recognition.

[0023] In view of this, this application provides an abnormal behavior recognition method and related device based on multimodal perception. The scheme first acquires visible light RGB video frame sequences and infrared video frame sequences of the target to be identified, and preprocesses these sequences. Based on the preprocessed visible light RGB video frame sequences and infrared video frame sequences, a preset first network is optimized to obtain an optimized first network. The preprocessed visible light RGB video frame sequences and infrared video frame sequences are input into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features. Preset abnormal behavior category text descriptions are encoded and aggregated to generate text classification weights. The similarity between the fused cross-modal visual features and the text classification weights is calculated to obtain the abnormal behavior classification probability. Then, the abnormal behavior of the target to be identified is determined based on the abnormal behavior classification probability. This application simultaneously introduces visible light and infrared dual-modal information, and effectively overcomes the limitations of single RGB modality in complex scenarios such as low light, backlight, nighttime, and occlusion by fusing the visible light and infrared dual-modal information across modalities. This improves the completeness of pedestrian behavior perception in complex environments and thus enhances the accuracy of abnormal behavior recognition. Furthermore, by using textual semantics as prior information for category semantics and combining it with the fused cross-modal features in the behavior discrimination process, visually similar but semantically different behaviors can be effectively distinguished with the help of prior textual semantic information, reducing the probability of confusion between similar behaviors and improving the accuracy and generalization ability of abnormal behavior discrimination in complex scenarios.

[0024] The abnormal behavior recognition method based on multimodal perception provided in this application relates to the field of behavior recognition technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the abnormal behavior recognition method based on multimodal perception, but is not limited to the above forms.

[0025] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0026] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided in an embodiment of this application. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0027] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0028] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0029] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment does not impose any limitations.

[0030] For example, based on Figure 1The implementation environment shown in this application embodiment provides an abnormal behavior recognition method based on multimodal perception. The following description uses the application of this abnormal behavior recognition method based on multimodal perception in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0031] Reference Figure 2 , Figure 2 The flowchart illustrates a method for identifying abnormal behavior on a server based on multimodal perception, provided in this application embodiment. The executing entity of this method can be any of the aforementioned computer devices (including a server or terminal). (Refer to...) Figure 2 The method may include the following steps: S100: Obtain the visible light RGB video frame sequence and the infrared video frame sequence of the target to be identified, and preprocess the visible light RGB video frame sequence and the infrared video frame sequence.

[0032] In this embodiment, a visible light camera and an infrared thermal imaging device are arranged in pairs on a support frame above the inspection channel. This allows the visible light camera to capture visible light RGB video frame sequences and the infrared thermal imaging device to capture infrared (IR) video frame sequences. Figure 3 The visible light RGB video frame image shown in (b) and Figure 3 (a) Mid-infrared video frame image.

[0033] Furthermore, the visible light RGB video frame sequence and the infrared video frame sequence are preprocessed, including: (1) Timestamp alignment: Since there are differences in sampling frequency, buffering mechanism and timestamp recording method between visible light cameras and infrared thermal imaging devices, the timestamp information corresponding to each frame in the visible light RGB video frame sequence and the infrared video frame sequence is read first; then the timestamp of the visible light RGB video frame at the current time t is used. For reference, search for the infrared video frame timestamp with the smallest absolute time difference in the infrared video frame sequence. A one-to-one correspondence is established between visible light RGB video frames and infrared video frames. The temporal mapping relationship can be expressed as:

[0034] in, Representation of time The IR frame index mapping function that best matches the visible light RGB video frames. This represents the Euclidean norm, used here to measure the difference between two timestamps. This represents an infrared video frame aligned with the visible light RGB video frame. This temporal alignment ensures that the visible light and infrared modalities remain consistent in the temporal dimension during subsequent cross-modal feature modeling.

[0035] (2) Sliding time window organization: To enhance the contextual expressiveness of behavioral patterns in the time dimension, a sliding time window with a length of [missing information] is used. The sliding time window organizes consecutive frames to form a visible light RGB video frame sequence and an aligned infrared video frame sequence, expressed as:

[0036] in, Indicates the current time The visible light RGB video frame sequence obtained by tracing back to the endpoint. This indicates the corresponding aligned infrared video frame sequence. Indicates the current time Forward RGB image of the frame This represents the IR (infrared) image aligned with the corresponding RGB image. Indicates the frame offset index within the time window. Indicates the length of the time window.

[0037] (3) Noise reduction: Denoise and smooth each RGB image in the visible light RGB video frame sequence to reduce ambient light disturbance and sensor noise; suppress thermal noise in each IR image in the infrared video frame sequence.

[0038] (4) Infrared pseudo-color encoding: Each frame of the infrared video frame sequence is converted into a three-channel pseudo-color image to match the RGB image in the channel dimension.

[0039] (5) Spatial scale normalization and key area cropping: Each RGB image and each IR image is uniformly scaled to the preset image size, and key areas are cropped according to the location of the port channel, gate area or waiting area to reduce irrelevant background interference.

[0040] (6) Normalize the pixel values ​​in each frame of RGB image and each frame of IR image.

[0041] (7) Stack consecutive frames in chronological order, and map the visible light RGB video frame sequence and infrared video frame sequence into RGB input tensors and infrared tensors with unified structure, respectively, to obtain the final processed visible light RGB video frame sequence and the preprocessed visible light RGB video frame sequence.

[0042] S200: The preset first network is optimized based on the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain the optimized first network.

[0043] In this embodiment, the first network is the student network, and its optimization process relies on a preset second network (i.e., the teacher network). This optimization is achieved through residual feature distillation, aligning the cross-modal visual features output by the student network with the feature distribution of the teacher network in the residual space, thereby improving the generalization ability and semantic consistency of the student network. For example, the steps of optimizing the preset first network based on preprocessed visible light RGB video frame sequences and preprocessed infrared video frame sequences to obtain the optimized first network include S210-S240: S210. Input the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into the preset first network for feature extraction to obtain the first visual feature of the visible light RGB video frame sequence and the second visual feature of the infrared video frame sequence. Specifically, the first network consists of a visual feature extraction backbone composed of multi-level parallel coding blocks connected in series. Each level of parallel coding block contains three paths: an identity path, a spatial attention path, and a temporal adaptation path. The identity path preserves the original feature distribution, preventing information from being excessively weakened during deep transmission. The spatial attention path, based on the frozen CLIP image encoder, spatially weights salient regions to highlight key targets and regions related to behavior. The temporal adaptation path models cross-frame motion information through 3D convolution or a temporal adapter, capturing the rhythm and trajectory changes of actions. Through this multi-path parallel coding method, independent representation learning is performed on the RGB and IR visual modalities to extract the first visual features of the RGB video frame sequence and the second visual features of the infrared video frame sequence, which contain spatial structure information, salient region information, and temporal dynamic information. This provides a stable and distinguishable basic feature representation for subsequent cross-modal interactive fusion.

[0044] S220. Input the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into a preset second network for feature extraction to obtain the third visual feature of the visible light RGB video frame sequence and the fourth visual feature of the infrared video frame sequence. For example, the second network is a teacher network that includes a learnable projector for a CLIP image encoder with frozen parameters. The teacher network extracts features from preprocessed visible light RGB video frame sequences and preprocessed infrared video frame sequences (wherein the IR frame sequences have been converted into three-channel images via pseudo-color encoding), respectively, to obtain high-quality third visual features for the visible light RGB video frame sequences and high-quality fourth visual features for the infrared video frame sequences.

[0045] S230. Determine the residual distillation loss based on the first visual feature, the second visual feature, the third visual feature, and the fourth visual feature; This embodiment constructs a residual distillation loss function to measure the difference between the visual features output by the student network and the high-quality visual features output by the teacher network.

[0046] S240. Optimize the parameters of the preset first network based on the residual distillation loss to obtain the optimized first network.

[0047] S300. The preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence are input into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features.

[0048] For example, the steps of inputting the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into the optimized first network for feature extraction and feature fusion processing to obtain the fused cross-modal visual features include S310-S330: S310. The optimized first network extracts features from the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain the fifth visual feature of the visible light RGB video frame sequence and the sixth visual feature of the infrared video frame sequence. S320. The fifth visual feature and the sixth visual feature are time-series weighted respectively to obtain the first weighted visual feature of the visible light RGB video frame sequence and the second weighted visual feature of the infrared video frame sequence. This application embodiment, in order to characterize the evolutionary characteristics of behavior in the time dimension, introduces a time-weighted operator to perform temporal weighting on the fifth visual feature and the sixth visual feature. The specific steps include S321-S324: S321. Calculate the first time weight of each first spatiotemporal feature unit based on the preset time decay coefficient and the index of each first spatiotemporal feature unit in the fifth visual feature. S322. Calculate the second time weight of each second spatiotemporal feature unit based on the preset time decay coefficient and the index of each second spatiotemporal feature unit in the sixth visual feature. For example, the first time weight and the second time weight in the embodiments of this application can be calculated by the following expression:

[0049] in, For the first Normalized time weights of spatiotemporal feature units This is a preset time decay coefficient used to control the importance of features at different temporal positions. and All are spatiotemporal feature unit indices.

[0050] S323. Weight each of the first spatiotemporal feature units according to the first time weight to obtain the first weighted visual feature of the visible light RGB video frame sequence. For example, the first weighted visual feature The expression is:

[0051] in, As the first time weight, The first in the visible light RGB video frame sequence The first spatiotemporal feature unit.

[0052] S324. Each of the second spatiotemporal feature units is weighted according to the second time weight to obtain the second weighted visual feature of the infrared video frame sequence.

[0053] For example, the second weighted visual feature The expression is: in, As the second time weight, The first in the infrared video frame sequence A second spatiotemporal feature unit.

[0054] S330. The first weighted visual feature and the second weighted visual feature are fused through a cross-attention mechanism to obtain fused cross-modal visual features; This application's embodiments employ a bidirectional cross-attention mechanism to model the complementary relationship between RGB and IR modes, and introduce a modality balance modulation factor for self-cross-modal visual feature fusion. The specific steps include S331-S337: S331. Map the first weighted visual feature to a first query matrix, and use the second weighted visual feature as the first key matrix and the first value matrix, respectively. For example, by querying the mapping matrix Key mapping matrix Sum value mapping matrix The first weighted visual feature Mapped to the first query matrix The second weighted visual feature Mapped to the first key matrix and the first value matrix ; where, query mapping matrix Key mapping matrix Sum value mapping matrix All of these are linear transformation parameters learned during network training.

[0055] S332. Calculate the similarity between the first query matrix and the first key matrix to obtain the first attention weight matrix, and normalize the first attention weight matrix. The normalized first attention weight matrix can be understood as representing the cross-modal attention weight matrix from the RGB mode to the IR mode; the normalized first attention weight matrix is ​​calculated. The expression is:

[0056] in, For feature dimension, This represents the transpose of the first key matrix. This represents the square root of the feature dimension, used to scale the dot product result to prevent the softmax output from being too sharp due to excessively large values.

[0057] S333. The normalized first attention weight matrix and the first value matrix are weighted and summed to obtain the first cross-modal interaction feature. The normalized first attention weight matrix and the first value matrix are weighted and summed to extract the infrared-supplemented features most relevant to the behavioral patterns of the current visible light RGB video frame sequence. The first weighted visual feature after infrared feature supplementation is then used as the first cross-modal interaction feature. Its expression is:

[0058] in, This represents the first cross-modal interaction feature formed by the RGB mode after absorbing supplementary information from the IR mode.

[0059] S334. Map the second weighted visual feature to the second query vector, and map the first weighted visual feature to the second key matrix and the second value matrix respectively; For example, by querying the mapping matrix Key mapping matrix Sum value mapping matrix The second weighted visual feature Mapped to the second query matrix The first weighted visual feature Mapped to the second bond matrix Second-value matrix .

[0060] S335. Calculate the similarity between the second query matrix and the second key matrix to obtain the second attention weight matrix, and normalize the second attention weight matrix. For example, the normalized second attention weight matrix can be understood as a cross-modal attention weight matrix representing the IR mode pointing to the RGB mode; the normalized second attention weight matrix is ​​calculated. The expression is:

[0061] in, For feature dimension, This represents the transpose of the first bond matrix. This represents the square root of the feature dimension, used to scale the dot product result to prevent the softmax output from being too sharp due to excessively large values.

[0062] S336. The normalized second attention weight matrix and the second value matrix are weighted and summed to obtain the second cross-modal interaction feature. The normalized second attention weight matrix and the second value matrix are weighted and summed to extract the visible light supplemented features most relevant to the behavioral patterns of the current infrared video frame sequence. The second weighted visual features supplemented with visible light features are then used as the second cross-modal interaction features. Its expression is:

[0063] S337. Based on the modal balance modulation factor, the first cross-modal interaction feature and the second cross-modal interaction feature are fused to obtain the fused cross-modal visual feature.

[0064] For example, to avoid a single mode dominating the fusion result, a mode balance modulation factor is introduced. The linear reconstruction of two cross-modal interaction features is expressed as follows:

[0065] in, This refers to the final fused cross-modal visual features, the modality balance modulation factor. This is used to adjust the proportions of the first cross-modal interaction feature (the feature after RGB mode absorbs IR mode supplementary information) and the second cross-modal interaction feature (the feature after IR mode absorbs RGB mode supplementary information) in the final fusion result, where, when When the value is large, more emphasis is placed on the heat source supplementary information from the IR mode; when When the size is smaller, more emphasis is placed on the original texture details of the RGB modality. The fused cross-modal visual features retain visible light detail information while introducing infrared thermal features to compensate for human body contours and movement patterns.

[0066] The embodiments of this application can establish fine-grained correspondence and constraints between RGB modal and IR modal features, and can simultaneously focus on action texture details and dynamic changes of heat sources, thereby obtaining a more discriminative feature behavior representation.

[0067] S400: Encode and aggregate the text descriptions of the preset abnormal behavior categories to generate text classification weights.

[0068] In this embodiment of the application, several abnormal behavior categories are predefined, such as "climbing over the turnstile," "tailgating passengers in front," "abnormal gathering," and "looking left and right." The text description corresponding to each abnormal behavior category is converted into a classification weight vector with semantic discriminative power. The steps include S410-S440: S410. Perform word segmentation on the preset behavior category text description to obtain a word element text sequence; S420. Encode the word text sequence to obtain a word text feature sequence; For example, in this embodiment of the application, the obtained lexical text sequence is input into a text encoder for encoding, thereby obtaining a lexical text feature sequence.

[0069] S430. Aggregate the word text features in the word text feature sequence to obtain an aggregated feature vector; S440. Normalize the aggregated feature vector to generate text classification weights.

[0070] This application uses the textual features of abnormal behavior as semantic priors to perform semantic modulation and weighting on visual features, so that the visual representation can be further aligned with the semantics of the target abnormal behavior category, thereby obtaining a high-dimensional behavioral feature representation under category semantic constraints, improving the ability to distinguish similar behaviors, ambiguous behaviors and complex scene behaviors, and reducing the probability of confusion between similar behaviors.

[0071] S500. Calculate the similarity between the fused cross-modal visual features and the text classification weights to obtain the abnormal behavior classification probability, and then determine the abnormal behavior of the target to be identified based on the abnormal behavior classification probability.

[0072] For example, the step of calculating the similarity between the fused cross-modal visual features and the text classification weights to obtain the abnormal behavior classification probability includes S510-S530: S510. Perform global average pooling on the fused cross-modal visual features to obtain a global feature vector; For example, global average pooling is performed on the fused cross-modal visual features to obtain a global feature vector. The expression is:

[0073] in, This represents the fused cross-modal visual feature corresponding to the nth spatiotemporal location.

[0074] S520. Perform a dot product operation between the global feature vector and each text classification weight in the preset abnormal behavior category to obtain the original similarity score of each abnormal behavior category. S530. The original similarity score for each abnormal behavior category is used to calculate the abnormal behavior classification probability through a category mapping function.

[0075] For example, the behavior belonging to the first category is calculated through the category mapping function. The expression for the classification probability of abnormal behavior is:

[0076] in, This represents the vector (i.e., the global feature vector) for a given behavior. Under the condition, belonging to the first The probability of abnormal behavior; Indicates the first The transpose of the classification weight vector corresponding to the class behavior; Indicates the first The transpose of the classification weight vector corresponding to the class behavior; Indicates the total number of behavior categories; This is the temperature adjustment coefficient, used to control the smoothness of the probability distribution.

[0077] In this application, the text semantics are used as a high-level prior for abnormal behavior categories and applied to visual features so that reliable recognition can still be achieved with the help of semantic conditions in scenarios where visual information is insufficient, action details are blurred, or parts are lost.

[0078] Furthermore, embodiments of this application can also construct a risk scoring function based on a predefined set of high-risk behaviors, the expression of which is:

[0079] in, This represents the overall risk score corresponding to the current behavior sample; This represents a set of high-risk behavior categories; Indicates the first The risk weight coefficient corresponding to each type of abnormal behavior is used to reflect the importance of different behaviors in the overall risk assessment.

[0080] Furthermore, risk scoring Risk levels are determined by comparing the data with a preset threshold range. When this happens, the system enters continuous monitoring mode; when When the system triggers an early warning, a warning message will be displayed. When this happens, the system initiates an active control and linkage response mechanism and writes relevant behavioral event information into the system's event storage module to support subsequent audit analysis and behavioral review.

[0081] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0082] In one specific embodiment, please refer to Figure 4 , Figure 4 This is a schematic diagram of the architecture of the abnormal behavior recognition method based on multimodal perception provided in the embodiments of this application, such as... Figure 4 As shown, Figure 4 In This represents a zero-sample semantic consistency constraint, which ensures that the fused visual features and the text semantic features corresponding to the real behavior categories remain consistent in a unified semantic space. This enables the model to improve its generalization ability to recognize complex behaviors, few-sample behaviors, or similar behaviors by leveraging text semantic priors. This represents the category score before softmax normalization; This represents the behavior category classification loss, used to supervise the model to correctly distinguish behavior categories such as climbing over, tailing, breaking through obstacles, gathering, and looking left and right. This represents the cross-modal consistency loss, used to constrain RGB modal features, IR modal features and the final fused features to maintain consistency in a unified feature space, in order to reduce semantic deviations caused by different modalities due to illumination, occlusion, thermal noise, sensor differences or viewpoint differences; This indicates event-level visual consistency loss; This represents the routing constraint loss, which is used to ensure that different paths in parallel coding blocks can adaptively participate in modeling based on input features, while avoiding the model's long-term reliance on a single path, which could cause other paths to fail. The residual feature distillation loss is used to constrain the alignment of the fused visual features of the student network output to the frozen visual prior features of the teacher network output. Specifically, the abnormal behavior recognition process of multimodal perception in this application includes the following steps: S1. Acquire video frame sequence: Visible light cameras and infrared thermal imaging devices are deployed in pairs on support frames above the port inspection channels to provide 24 / 7 monitoring of key areas such as manual counters, self-service gates, and waiting areas. The visible light cameras capture high-resolution visible light RGB video frame sequences to obtain information on human posture, texture, and appearance; the infrared thermal imaging devices capture infrared video frame sequences (IR video frame sequences) to obtain the heat source distribution and contour structure of the human body and its surrounding environment, providing stable supplementary information under complex lighting conditions such as low light, backlight, and nighttime.

[0083] S2. Preprocess the video frame sequence, including: 1. Timestamp Alignment: Due to differences in sampling frequency, buffering mechanism, and timestamp recording method between visible light cameras and infrared thermal imaging devices, the timestamp information corresponding to each frame in the visible light RGB video frame sequence and the infrared video frame sequence is first read separately; then, the timestamp of the visible light RGB video frame at the current time t is used as the basis for alignment. For reference, search for the infrared video frame timestamp with the smallest absolute time difference in the infrared video frame sequence. A one-to-one correspondence is established between visible light RGB video frames and infrared video frames. The temporal mapping relationship can be expressed as:

[0084] in, Representation of time The IR frame index mapping function that best matches the visible light RGB video frames. This represents the Euclidean norm, used here to measure the difference between two timestamps. This represents an infrared video frame aligned with the visible light RGB video frame. This temporal alignment ensures that the visible light and infrared modes remain consistent in the temporal dimension during subsequent cross-modal feature modeling.

[0085] 2. Sliding Time Window Organization: To enhance the contextual expressiveness of behavioral patterns across time, a sliding time window with a length of [missing information] is used. The sliding time window organizes consecutive frames to form a visible light RGB video frame sequence and an aligned infrared video frame sequence, expressed as:

[0086] in, Indicates the current time The visible light RGB video frame sequence obtained by tracing back to the endpoint. This indicates the corresponding aligned infrared video frame sequence. Indicates the current time Forward RGB image of the frame This represents the IR (infrared) image aligned with the corresponding RGB image. Indicates the frame offset index within the time window. Indicates the length of the time window.

[0087] 3. Noise Reduction Processing: Denoise and smooth each RGB image in the visible light RGB video frame sequence to reduce ambient light disturbances and sensor noise; suppress thermal noise in each IR image in the infrared video frame sequence.

[0088] 4. Infrared pseudo-color encoding: Each frame of the infrared video frame sequence is converted into a three-channel pseudo-color image to match the RGB image in the channel dimension.

[0089] 5. Spatial scale normalization and key area cropping: Each RGB image and each IR image is uniformly scaled to a preset image size, and key areas are cropped according to the location of the port channel, gate area or waiting area to reduce irrelevant background interference.

[0090] 6. Normalize the pixel values ​​in each frame of RGB image and each frame of IR image.

[0091] 7. Stack consecutive frames in chronological order, and map the visible light RGB video frame sequence and infrared video frame sequence into RGB input tensors and infrared tensors with unified structures, respectively, to obtain the final processed visible light RGB video frame sequence and the preprocessed visible light RGB video frame sequence.

[0092] S3. Extract visual features from the visible light RGB and infrared (IR) modalities: In this embodiment, an optimized student network is used to extract visual features in two modalities: visible light RGB and infrared (IR). Specifically, the optimized student network includes: inputting the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence from step S2 into the teacher network to extract high-quality visual features in both visible light RGB and infrared (IR) modalities; constructing a residual distillation loss function based on the high-quality visual features of the two modalities and the visual features extracted by the student network before optimization; calculating the residual distillation loss; and then optimizing the student network based on the residual distillation loss to obtain the optimized student network.

[0093] Furthermore, the student network routing uses a multi-level parallel coding block concatenated to form the visual feature extraction backbone. Each level of parallel coding block contains three sub-paths: an identity path, a spatial attention path, and a temporal adaptation path. The identity path preserves the original feature distribution, preventing excessive information attenuation during deep transmission. The spatial attention path, based on the frozen CLIP image encoder, spatially weights salient regions to highlight key targets and regions relevant to behavior. The temporal adaptation path models cross-frame motion information through 3D convolution or a temporal adapter, capturing the rhythm and trajectory changes of actions. Through this multi-path parallel coding method, independent representation learning is performed on both RGB and IR visual modalities to extract visual features from RGB video frame sequences and infrared video frame sequences, respectively, which possess spatial structure information, salient region information, and temporal dynamic information. The expressions are as follows: .

[0094] in, This represents the visual features extracted from the RGB modality. This indicates the encoding module corresponding to RGB. This represents the preprocessed visible light RGB video frame sequence. This represents the visual features obtained from IR modality extraction. This indicates the encoding module corresponding to the IR. This represents the preprocessed IR video frame sequence. Indicates the number of spatiotemporal feature units. Indicates the feature embedding dimension.

[0095] S4, Cross-modal visual feature fusion: First, the visual features of the RGB mode and the visual features of the IR mode extracted from the optimized student network are temporally weighted to obtain the first weighted visual features of the visible light RGB video frame sequence. Second weighted visual features of infrared video frame sequences Its expression is: ,

[0096] in, The first in the visible light RGB video frame sequence The first spatiotemporal feature unit, The first time weight is the time weight corresponding to the RGB video frame sequence. The first in the infrared video frame sequence The second spatiotemporal feature unit is the temporal weight of the infrared video frame sequence; The time weight can be calculated using the following expression:

[0097] in, For the first Normalized time weights of spatiotemporal feature units This is a preset time decay coefficient used to control the importance of features at different temporal positions. and All are spatiotemporal feature unit indices.

[0098] Secondly, the first weighted visual features and the second weighted visual features are fused through a cross-attention mechanism to obtain fused cross-modal visual features. This includes: constructing a query matrix using the first weighted visual features, constructing a key matrix and a value matrix using the second weighted visual features, and establishing an RGB→IR cross-attention mechanism. This allows the first weighted visual features (RGB modality) to actively retrieve and fuse heat source contours, temperature distribution, and weak light supplementation information related to the current behavior in the IR modality during the attention weighting process, thus obtaining cross-modal interaction features in the RGB→IR interaction process. Subsequently, a query matrix is ​​constructed using the second weighted visual features (IR), and a key matrix and a value matrix are constructed using the RGB features. This establishes an IR→RGB reverse cross-attention mechanism, allowing the second weighted features to further incorporate texture details, target appearance, and spatial boundary information from the RGB modality, thus obtaining cross-modal interaction features in the IR→RGB interaction process. A modal balance modulation factor is further introduced to linearly reconstruct the bidirectional cross-modal interaction features, resulting in fused cross-modal visual features, the expression of which is:

[0099] in, This refers to the final fused cross-modal visual features, the modality balance modulation factor. This is used to adjust the proportions of the first cross-modal interaction feature (the feature after RGB mode absorbs IR mode supplementary information) and the second cross-modal interaction feature (the feature after IR mode absorbs RGB mode supplementary information) in the final fusion result, where, when When the value is large, more emphasis is placed on the heat source supplementary information from the IR mode; when When the size is smaller, more emphasis is placed on the original texture details of the RGB modality. The fused cross-modal visual features retain visible light detail information while introducing infrared thermal features to compensate for human body contours and movement patterns.

[0100] S5. Generate text classification weights from the text descriptions of abnormal behavior categories: First, the behavioral category text descriptions (such as "climbing over the turnstile", "following the passenger in front", "abnormal gathering", "looking left and right") are segmented to obtain a word text sequence. The word text sequence is then input into a text encoder for encoding to obtain a word text feature sequence. Second, the word text features in the word text feature sequence are aggregated. Finally, the aggregated feature vectors are normalized to generate text classification weights. S6. Identify abnormal behavior of the target: First, perform global average pooling in the spatiotemporal direction on the fused cross-modal visual features to obtain a global feature vector; second, perform a dot product operation between the global feature vector and the text classification weights in the preset abnormal behavior categories to obtain the original similarity score of each abnormal behavior category; finally, calculate the abnormal behavior classification probability by using the category mapping function on the original similarity score, and the abnormal behavior category can be identified based on the abnormal behavior classification probability.

[0101] For example, calculating the behavior belonging to the first category using a category mapping function. The expression for the classification probability of abnormal behavior is:

[0102] in, This represents the vector (i.e., the global feature vector) for a given behavior. Under the condition, belonging to the first The probability of abnormal behavior; Indicates the first The transpose of the classification weight vector corresponding to the class behavior; Indicates the first The transpose of the classification weight vector corresponding to the class behavior; Indicates the total number of behavior categories; This is a temperature adjustment coefficient used to control the smoothness of the probability distribution.

[0103] In addition, a risk scoring function can be constructed based on a predefined set of high-risk behaviors, the expression of which is:

[0104] in, This represents the overall risk score corresponding to the current behavior sample; This represents a set of high-risk behavior categories; Indicates the first The risk weight coefficient corresponding to each type of abnormal behavior is used to reflect the importance of different behaviors in the overall risk assessment.

[0105] Furthermore, risk scoring Risk levels are determined by comparing the data with a preset threshold range. When this happens, the system enters continuous monitoring mode; when When a behavior is identified as having a mild to moderate risk (such as abnormal gathering or prolonged stay), the system triggers an early warning. This involves pushing a notification to the backend management terminal and recording elements such as the channel number, time of occurrence, and behavior category for subsequent trend analysis and management decisions. When a high-risk behavior is identified, the system activates an active control and linkage response mechanism. This involves the system linking the gate controller and audible / visual alarm devices to implement channel blocking and on-site warnings. The relevant behavioral event information is written to the system's event storage module to support subsequent audit analysis and behavior review. The identification results of this application can directly link on-site audible / visual alarm devices and channel control devices, achieving an automatic closed loop from "identification" to "intervention." For high-risk behaviors such as climbing over barriers or forcibly breaking through, the system can trigger an intervention-level response within milliseconds, completing actions such as audible / visual warnings, gate blocking, and pedestrian guidance, effectively compressing the safety handling time window. All identified events are automatically recorded in a structured manner, including key data such as behavior type, occurrence time, location, duration, participating parties, and multimodal confidence level, and are organized and stored in the form of event chains, providing managers with a complete behavioral trajectory and contextual information. Based on this data, further statistical analysis, pattern mining, and strategy optimization can be carried out.

[0106] Furthermore, the abnormal behavior recognition system architecture in this application supports modular assembly and flexible scheduling of heterogeneous computing resources, allowing for differentiated configurations based on different positions and channel types. The system can be centrally deployed on a central server as a high-performance GPU cluster, uniformly accessing data streams from multiple cameras and infrared devices to handle deep inference and global policy management; alternatively, lightweight models and key decision logic can be deployed to edge computing terminals for rapid analysis and immediate alerts of video and infrared data, reducing network transmission load and shortening response latency. Leveraging core technologies such as multimodal fusion, text semantic guidance, and residual feature distillation, this solution can continuously output stable and reliable intelligent recognition and risk decision-making results in complex application scenarios such as high-density passage, rapid customs clearance, and human-machine collaborative management, meeting the practical needs of smart ports for refined management and dynamic risk control.

[0107] In summary, this application provides a method and related device for abnormal behavior recognition based on multimodal perception. The scheme first acquires visible light RGB video frame sequences and infrared video frame sequences of the target to be identified, and preprocesses these sequences. Based on the preprocessed visible light RGB video frame sequences and infrared video frame sequences, a preset first network is optimized to obtain an optimized first network. The preprocessed visible light RGB video frame sequences and infrared video frame sequences are input into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features. Preset abnormal behavior category text descriptions are encoded and aggregated to generate text classification weights. The similarity between the fused cross-modal visual features and the text classification weights is calculated to obtain the abnormal behavior classification probability. Finally, the abnormal behavior of the target to be identified is determined based on the abnormal behavior classification probability. This application simultaneously introduces visible light and infrared dual-modal information, and effectively overcomes the limitations of single RGB modality in complex scenarios such as low light, backlight, nighttime, and occlusion by fusing the visible light and infrared dual-modal information across modalities. This improves the completeness of pedestrian behavior perception in complex environments and thus enhances the accuracy of abnormal behavior recognition. Furthermore, by using textual semantics as prior information for category semantics and combining it with the fused cross-modal features in the behavior discrimination process, visually similar but semantically different behaviors can be effectively distinguished with the help of prior textual semantic information, reducing the probability of confusion between similar behaviors and improving the accuracy and generalization ability of abnormal behavior discrimination in complex scenarios.

[0108] like Figure 5 As shown in the figure, this application also provides a structural schematic diagram of an abnormal behavior recognition device based on multimodal perception. This device can implement the above-described method and may include: The sequence preprocessing module 21 is used to acquire the visible light RGB video frame sequence and the infrared video frame sequence of the target to be identified, and to preprocess the visible light RGB video frame sequence and the infrared video frame sequence. Network optimization module 22 is used to optimize a preset first network based on the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain an optimized first network; The feature fusion module 23 is used to input the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features; The classification weight generation module 24 is used to encode and aggregate the text descriptions of preset abnormal behavior categories to generate text classification weights; The abnormal behavior determination module 25 is used to calculate the similarity between the fused cross-modal visual features and the text classification weights to obtain the abnormal behavior classification probability, and then determine the abnormal behavior of the target to be identified based on the abnormal behavior classification probability.

[0109] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0110] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described abnormal behavior recognition method based on multimodal perception. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0111] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0112] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to implement the abnormal behavior recognition method based on multimodal perception of the embodiments of this application. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0113] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described abnormal behavior recognition method based on multimodal perception.

[0114] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0115] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0116] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0117] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0121] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An abnormal behavior recognition method based on multimodal perception, characterized in that, include: Acquire the visible light RGB video frame sequence and the infrared video frame sequence of the target to be identified, and preprocess the visible light RGB video frame sequence and the infrared video frame sequence; The preset first network is optimized based on the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain the optimized first network; The preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence are input into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features; The text descriptions of predefined abnormal behavior categories are encoded and aggregated to generate text classification weights; The similarity between the fused cross-modal visual features and the text classification weights is calculated to obtain the abnormal behavior classification probability, and then the abnormal behavior of the target to be identified is determined based on the abnormal behavior classification probability.

2. The abnormal behavior recognition method based on multimodal perception according to claim 1, characterized in that, The optimization of the preset first network based on the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence yields an optimized first network, including: The preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence are input into a preset first network for feature extraction to obtain the first visual feature of the visible light RGB video frame sequence and the second visual feature of the infrared video frame sequence. The preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence are input into a preset second network for feature extraction to obtain the third visual feature of the visible light RGB video frame sequence and the fourth visual feature of the infrared video frame sequence. The residual distillation loss is determined based on the first visual feature, the second visual feature, the third visual feature, and the fourth visual feature; The parameters of the preset first network are optimized based on the residual distillation loss to obtain the optimized first network.

3. The abnormal behavior recognition method based on multimodal perception according to claim 1, characterized in that, The step of inputting the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features includes: The optimized first network extracts features from the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain the fifth visual feature of the visible light RGB video frame sequence and the sixth visual feature of the infrared video frame sequence. The fifth visual feature and the sixth visual feature are time-weighted respectively to obtain the first weighted visual feature of the visible light RGB video frame sequence and the second weighted visual feature of the infrared video frame sequence; The first weighted visual feature and the second weighted visual feature are fused by a cross-attention mechanism to obtain a fused cross-modal visual feature.

4. The abnormal behavior recognition method based on multimodal perception according to claim 3, characterized in that, The step of performing temporal weighting on the fifth visual feature and the sixth visual feature respectively to obtain the first weighted visual feature of the visible light RGB video frame sequence and the second weighted visual feature of the infrared video frame sequence includes: The first time weight of each first spatiotemporal feature unit is calculated based on the preset time decay coefficient and the index of each first spatiotemporal feature unit in the fifth visual feature. The second time weight of each second spatiotemporal feature unit is calculated based on the preset time decay coefficient and the index of each second spatiotemporal feature unit in the sixth visual feature. Each of the first spatiotemporal feature units is weighted according to the first time weight to obtain the first weighted visual feature of the visible light RGB video frame sequence. Each of the second spatiotemporal feature units is weighted according to the second time weight to obtain the second weighted visual feature of the infrared video frame sequence.

5. The abnormal behavior recognition method based on multimodal perception according to claim 3, characterized in that, The process of fusing the first weighted visual features and the second weighted visual features through a cross-attention mechanism to obtain fused cross-modal visual features includes: The first weighted visual feature is mapped to a first query matrix, and the second weighted visual feature is mapped to a first key matrix and a first value matrix, respectively. Calculate the similarity between the first query matrix and the first key matrix to obtain the first attention weight matrix, and then normalize the first attention weight matrix. The first attention weight matrix after normalization is summed with the first value matrix to obtain the first cross-modal interaction feature. The second weighted visual feature is mapped to the second query matrix, and the first weighted visual feature is mapped to the second key matrix and the second value matrix, respectively. Calculate the similarity between the second query matrix and the second key matrix to obtain the second attention weight matrix, and then normalize the second attention weight matrix. The second attention weight matrix and the second value matrix are weighted and summed to obtain the second cross-modal interaction feature; Based on the modal balance modulation factor, the first cross-modal interaction feature and the second cross-modal interaction feature are fused to obtain the fused cross-modal visual feature.

6. The abnormal behavior recognition method based on multimodal perception according to claim 1, characterized in that, The process of encoding and aggregating the text descriptions of preset abnormal behavior categories to generate text classification weights includes: The text descriptions of preset behavior categories are segmented to obtain a word-based text sequence. The word text sequence is encoded to obtain a word text feature sequence; Aggregate the word text features in the word text feature sequence to obtain an aggregated feature vector; The aggregated feature vector is normalized to generate text classification weights.

7. The abnormal behavior recognition method based on multimodal perception according to claim 1, characterized in that, The calculation of the similarity between the fused cross-modal visual features and the text classification weights to obtain the abnormal behavior classification probability includes: Global average pooling is performed on the fused cross-modal visual features to obtain a global feature vector; The global feature vector is multiplied by the text classification weight in the preset abnormal behavior category to obtain the original similarity score of each abnormal behavior category. The original similarity score for each abnormal behavior category is used to calculate the abnormal behavior classification probability through a category mapping function.

8. An abnormal behavior recognition device based on multimodal perception, characterized in that, The device includes: The sequence preprocessing module is used to acquire the visible light RGB video frame sequence and the infrared video frame sequence of the target to be identified, and to preprocess the visible light RGB video frame sequence and the infrared video frame sequence. The network optimization module is used to optimize the preset first network based on the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence to obtain the optimized first network. The feature fusion module is used to input the preprocessed visible light RGB video frame sequence and the preprocessed infrared video frame sequence into the optimized first network for feature extraction and feature fusion processing to obtain fused cross-modal visual features; The classification weight generation module is used to encode and aggregate the text descriptions of preset abnormal behavior categories to generate text classification weights. An abnormal behavior determination module is used to calculate the similarity between the fused cross-modal visual features and the text classification weights to obtain the abnormal behavior classification probability, and then determine the abnormal behavior of the target to be identified based on the abnormal behavior classification probability.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the abnormal behavior recognition method based on multimodal perception as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the abnormal behavior recognition method based on multimodal perception as described in any one of claims 1 to 7.

Images