A method and system for recognizing abnormal behavior based on multi-modal information fusion

Abstract

The application discloses a method and system for identifying abnormal behavior based on multi-modal information fusion, and belongs to the technical field of information fusion. The method comprises the following steps: acquiring the joint position information of a target person in a target scene; prompting the identification model to determine the human behavior characteristics of the to-be-identified person based on the space-time change information of the joints of the to-be-identified person; prompting the database to determine the human vital signs of the to-be-identified person based on the human vital sign information of the to-be-identified person; performing multi-modal information fusion on the human behavior characteristics and the human vital signs of the to-be-identified person to generate a data set for determining the state of the to-be-identified person, and determining whether the state of the to-be-identified person is abnormal based on the data set. The application can identify the abnormal behavior of the person in the target scene, can identify the behavior of the caregiver, can determine the behavior state of the caregiver in time, and can timely issue an alarm for the abnormal behavior state of the caregiver.

Description

Technical Field

[0001] This invention relates to the field of information fusion technology, and more specifically, to a method and system for identifying abnormal behavior based on multimodal information fusion. Background Technology

[0002] With increased global government investment in smart cities, the smart surveillance industry continues to grow, projected to increase from $257.9 billion in 2019 to $397.6 billion in 2024. China's market will become a significant source of growth for the global security surveillance market. As video surveillance technology upgrades, moving from simply seeing and seeing further to seeing clearly, it brings with it reform and innovation: the emergence of intelligent monitoring further enables people to "understand what they see." The integration of modern information systems with "smart" and "cloud" technologies has become the mainstream solution in the industry, driving the upgrading and replacement of existing front-end and back-end hardware and software, as well as new construction demand, opening up growth space for the industry. Identifying and analyzing the behavior of monitored targets is a key step in intelligent monitoring.

[0003] Currently, common behavior recognition analysis is mainly based on deep learning methods, divided into two-stage detection and single-stage detection. Two-stage detection algorithms first generate detection regions, then classify and calibrate them. In 2015, scholars proposed the Faster R-CNN algorithm, pioneering deep learning-based two-stage detection. In 2017, the original authors improved upon this by replacing the dedicated candidate window generation algorithm with RPN, generating high-quality region proposals through end-to-end training. In 2017, scholars proposed Mask R-CNN, replacing the RoI pooling layer in the original Faster R-CNN with a RoI alignment layer, enabling more accurate alignment between original image pixels and feature maps. To address the inefficiency of two-stage detection, scholars proposed a single-stage object detector, YOLO. Subsequently, a series of improved YOLO algorithms were introduced. Taking the latest YOLOv5 as an example, the image passes through four parts: input, backbone, neck, and prediction, generating bounding boxes, predicting categories, and performing automatic calibration. The detection speed is very fast, with an inference time of 0.007 seconds per image.

[0004] Deep learning methods applied to human behavior recognition require utilizing spatial and temporal information from videos. Based on the characteristics of the backbone network, deep learning-based action recognition and behavior understanding networks mainly include: Two-Stream Convolutional Networks and 3D-ConvNets.

[0005] Two-Stream methods effectively extract appearance and motion information from video behavior recognition tasks, but they cannot effectively utilize the spatiotemporal information in videos. Later, researchers proposed a human behavior recognition method based on an improved two-stream spatiotemporal network. This method utilizes convolutional neural networks to construct a two-stream network structure containing both spatiotemporal motion and appearance information streams, improving the ability to identify similar behaviors and behaviors with strong temporal dependencies. Three-dimensional convolutional networks extract spatiotemporal features from videos through 3D convolutional kernels, capturing motion information from video streams. To address the shortcomings of 3D-CNNs, such as their inability to fully exploit long-term temporal information and the need for large-scale labeled datasets, researchers proposed an improved crowd abnormal behavior detection method combining an improved C3D network with a random forest. This method reduces the input dimensionality by using HOG features and replaces the SoftMax fully connected layer in the initial 3D convolutional network with a random forest classifier, reducing the required training sample size.

[0006] While the above methods can effectively identify the behavior of target individuals, they do not take into account the characteristics of high-resolution data in practical applications, such as large number of parameters and high computational complexity. Furthermore, using single video data for behavior analysis has the disadvantage of limited behavior recognition range. Summary of the Invention

[0007] To address the above problems, this invention proposes a method for identifying abnormal behavior based on multimodal information fusion, comprising:

[0008] The system acquires feature information of the target scene, establishes a description sentence generation model based on the feature information of the target scene using a convolutional neural network, monitors the target scene using the description sentence generation model, acquires the spatiotemporal change information of the key points of the target person in the target scene, and monitors the target scene using a non-contact sensor to acquire the key point position information of the target person in the target scene.

[0009] An extraction model for extracting human behavioral features is established based on the motion features of the target person, and the extraction model is lightweighted to generate a recognition model. The target scene is monitored through the model generated by the description statement to obtain the spatiotemporal change information of the key points of the person to be identified, and the spatiotemporal change information of the key points of the person to be identified is sent to the recognition model to enable the recognition model to determine the human behavioral features of the person to be identified based on the spatiotemporal change information of the key points of the person to be identified.

[0010] A database for determining human physical characteristics is established based on the joint position information of the target person. The target scene is monitored by non-contact sensors to obtain the joint position information of the person to be identified and send the joint position information to the database so that the database can determine the human physical characteristics of the person to be identified based on the joint position information of the person to be identified.

[0011] Multimodal information fusion is performed on the human behavioral features and physical characteristics of the person to be identified to generate a dataset for determining the state of the person to be identified, and the state of the person to be identified is determined based on the dataset.

[0012] The multimodal information fusion includes: establishing a waveform time series of the state of the person to be identified based on the human behavioral characteristics and physical signs of the person to be identified, converting the waveform time series into image data, and generating a dataset based on the image data.

[0013] Optionally, the target scene is monitored using the model generated by the description statement to obtain spatiotemporal change information of the key points of the target person within the target scene, including:

[0014] The target scene is monitored based on the description statement generation model to obtain image data of the target person in the target scene. The image data is then segmented to obtain the human foreground feature map of the target person. The region of interest image data in the human foreground feature map is determined. The image data and the region image data are then input into the pooling layer of the description statement generation model. The pooling layer detects multiple key points of the target person to extract the spatiotemporal change information of the target person's joints.

[0015] Optionally, the model can be extracted, including various processing modules based on convolutional neural networks;

[0016] The processing module includes at least one of the following: a multi-scale image preprocessing module, an inverse residual module, an attention fusion module, a Gattneck basic module, a Gattblock basic module, and a Gattnet basic module.

[0017] Optionally, the extraction model is lightweighted, including modifying the structure of the multi-scale image preprocessing module, the Gattneck basic module, and the Gattblock basic module, including:

[0018] The modification to the structure of the multi-scale image preprocessing module includes: replacing the 3×3 convolution submodule of the multi-scale image preprocessing module with a lightweight 1×1 depth convolution submodule, and adding a 3×3 transposed convolution submodule to the multi-scale image processing module to perform transposed convolution by addition.

[0019] The structures of the Gattneck and Gattblock basic modules are modified, including: fusing the Gattneck and Gattblock basic modules with the attention fusion module respectively, adding the InteAttention sub-module to the Gattneck and Gattblock basic modules, and replacing the 3×3 operator of the Gattneck basic module with a 1×1 lightweight operator.

[0020] Furthermore, this invention also provides a system for identifying abnormal behavior based on multimodal information fusion, comprising:

[0021] The initial unit is used to acquire feature information of the target scene, establish a description sentence generation model based on the feature information of the target scene using a convolutional neural network, monitor the target scene through the description sentence generation model, acquire the spatiotemporal change information of the key points of the target person in the target scene, and monitor the target scene through a non-contact sensor to acquire the key point position information of the target person in the target scene.

[0022] The first recognition module is used to establish an extraction model for extracting human behavior features based on the motion features of the target person, and to perform lightweight processing on the extraction model to generate a recognition model. The model is generated by the description statement to monitor the target scene, obtain the spatiotemporal change information of the key points of the person to be identified, and send the spatiotemporal change information of the key points of the person to be identified to the recognition model so that the recognition model can determine the human behavior features of the person to be identified based on the spatiotemporal change information of the key points of the person to be identified.

[0023] The second identification module is used to establish a database for determining human physical characteristics based on the joint position information of the target person. It monitors the target scene through a non-contact sensor, obtains the joint position information of the person to be identified, and sends the joint position information to the database so that the database can determine the human physical characteristics of the person to be identified based on the joint position information of the person to be identified.

[0024] The fusion unit is used to perform multimodal information fusion on the human behavioral features and physical characteristics of the person to be identified in order to generate a dataset for determining the state of the person to be identified, and to determine whether the state of the person to be identified is abnormal based on the dataset.

[0025] The multimodal information fusion includes: establishing a waveform time series of the state of the person to be identified based on the human behavioral characteristics and physical signs of the person to be identified, converting the waveform time series into image data, and generating a dataset based on the image data.

[0026] Optionally, the description statement generation model monitors the target scene and obtains spatiotemporal change information of key points of the target person within the target scene, including:

[0027] The target scene is monitored based on the description statement generation model to obtain image data of the target person in the target scene. The image data is then segmented to obtain the human foreground feature map of the target person. The region of interest image data in the human foreground feature map is determined. The image data and the region image data are then input into the pooling layer of the description statement generation model. The pooling layer detects multiple key points of the target person to extract the spatiotemporal change information of the target person's joints.

[0028] Optionally, the model can be extracted, including various processing modules based on convolutional neural networks;

[0029] The processing module includes at least one of the following: a multi-scale image preprocessing module, an inverse residual module, an attention fusion module, a Gattneck basic module, a Gattblock basic module, and a Gattnet basic module.

[0030] Optionally, the extraction model is lightweighted, including modifying the structure of the multi-scale image preprocessing module, the Gattneck basic module, and the Gattblock basic module, including:

[0031] The modification to the structure of the multi-scale image preprocessing module includes: replacing the 3×3 convolution submodule of the multi-scale image preprocessing module with a lightweight 1×1 depth convolution submodule, and adding a 3×3 transposed convolution submodule to the multi-scale image processing module to perform transposed convolution by addition.

[0032] The structures of the Gattneck and Gattblock basic modules are modified, including: fusing the Gattneck and Gattblock basic modules with the attention fusion module respectively, adding the InteAttention sub-module to the Gattneck and Gattblock basic modules, and replacing the 3×3 operator of the Gattneck basic module with a 1×1 lightweight operator.

[0033] In another aspect, the present invention also provides a computing device, comprising: one or more processors;

[0034] A processor is used to execute one or more programs;

[0035] When the one or more programs are executed by the one or more processors, the method described above is implemented.

[0036] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described above.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] This invention provides a method for identifying abnormal behavior based on multimodal information fusion, comprising: acquiring feature information of a target scene; establishing a descriptive statement generation model based on the feature information of the target scene using a convolutional neural network; monitoring the target scene using the descriptive statement generation model to acquire spatiotemporal change information of the key points of a target person within the target scene; simultaneously monitoring the target scene using a non-contact sensor to acquire the position information of the key points of the target person within the target scene; establishing an extraction model for extracting human behavior features from the motion features of the target person, and performing lightweight processing on the extraction model to generate a recognition model; monitoring the target scene using the descriptive statement generation model to acquire spatiotemporal change information of the key points of the person to be identified, and sending the spatiotemporal change information of the key points of the person to be identified to the recognition model to prompt the recognition model to identify the person based on the descriptive statement generation model. The invention identifies the human behavioral characteristics of a person by analyzing the spatiotemporal changes of their joints. A database for determining human physical characteristics is established based on the joint position information of the target person. The target scene is monitored using non-contact sensors to acquire the joint position information of the person to be identified, and this information is sent to the database. The database then determines the human physical characteristics of the person to be identified based on this information. Multimodal information fusion is performed on the human behavioral characteristics and physical characteristics of the person to be identified to generate a dataset for determining the state of the person to be identified. Based on this dataset, it is determined whether the state of the person to be identified is abnormal. The multimodal information fusion includes: establishing a waveform time series of the state of the person to be identified based on their human behavioral characteristics and physical characteristics, converting the waveform time series into image data, and generating a dataset based on the image data. This invention can identify abnormal behavior of people in a target scene, enabling the identification of caregivers' behavior and timely determination of their behavioral state, and issuing timely alarms for abnormal behavior. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention;

[0040] Figure 2 This is a schematic diagram illustrating the principle of the method of the present invention for obtaining the motion characteristics of a target person within a target scene;

[0041] Figure 3This is a structural diagram of the system of the present invention. Detailed Implementation

[0042] Exemplary embodiments of the invention will now be described with reference to the accompanying drawings. However, the invention may be embodied in many different forms and is not limited to the embodiments described herein. These embodiments are provided to fully and completely disclose the invention and to fully convey its scope to those skilled in the art. The terminology used in the exemplary embodiments illustrated in the drawings is not intended to limit the invention. In the drawings, the same units / elements are referred to by the same reference numerals.

[0043] Unless otherwise stated, the terms used herein (including technical terms) have their common meaning as understood by one of ordinary skill in the art. Furthermore, it is understood that terms defined in commonly used dictionaries should be understood to have a meaning consistent with the context of their relevant field, and not to be interpreted as having an idealized or overly formal meaning.

[0044] Example 1:

[0045] The invention proposes a method for identifying abnormal behavior based on multimodal information fusion, such as... Figure 1 As shown, it includes:

[0046] Step 1: Obtain feature information of the target scene. Based on the convolutional neural network, establish a description sentence generation model using the feature information of the target scene. Monitor the target scene through the description sentence generation model to obtain the spatiotemporal change information of the key points of the target person in the target scene. At the same time, monitor the target scene through non-contact sensors to obtain the key point position information of the target person in the target scene.

[0047] Step 2: Establish an extraction model for extracting human behavior features based on the motion features of the target person, and perform lightweight processing on the extraction model to generate a recognition model. The model is generated by the description statement to monitor the target scene, obtain the spatiotemporal change information of the key points of the person to be identified, and send the spatiotemporal change information of the key points of the person to be identified to the recognition model so that the recognition model can determine the human behavior features of the person to be identified based on the spatiotemporal change information of the key points of the person to be identified.

[0048] Step 3: Establish a database for determining human physical characteristics based on the joint position information of the target person. Monitor the target scene using non-contact sensors to obtain the joint position information of the person to be identified, and send the joint position information to the database so that the database can determine the human physical characteristics of the person to be identified based on the joint position information of the person to be identified.

[0049] Step 4: Perform multimodal information fusion on the human behavioral features and physical characteristics of the person to be identified to generate a dataset for determining the state of the person to be identified, and determine whether the state of the person to be identified is abnormal based on the dataset.

[0050] The multimodal information fusion includes: establishing a waveform time series of the state of the person to be identified based on the human behavioral characteristics and physical signs of the person to be identified, converting the waveform time series into image data, and generating a dataset based on the image data.

[0051] The process involves monitoring the target scene using the description statement generation model to obtain spatiotemporal change information of key points of the target character within the target scene, including:

[0052] The target scene is monitored based on the description statement generation model to obtain image data of the target person in the target scene. The image data is then segmented to obtain the human foreground feature map of the target person. The region of interest image data in the human foreground feature map is determined. The image data and the region image data are then input into the pooling layer of the description statement generation model. The pooling layer detects multiple key points of the target person to extract the spatiotemporal change information of the target person's joints.

[0053] The extraction model includes various processing modules based on convolutional neural networks;

[0054] The processing module includes at least one of the following: a multi-scale image preprocessing module, an inverse residual module, an attention fusion module, a Gattneck basic module, a Gattblock basic module, and a Gattnet basic module.

[0055] The extraction model is lightweighted by modifying the structure of the multi-scale image preprocessing module, the Gattneck basic module, and the Gattblock basic module, including:

[0056] The modification to the structure of the multi-scale image preprocessing module includes: replacing the 3×3 convolution submodule of the multi-scale image preprocessing module with a lightweight 1×1 depth convolution submodule, and adding a 3×3 transposed convolution submodule to the multi-scale image processing module to perform transposed convolution by addition.

[0057] The structures of the Gattneck and Gattblock basic modules are modified, including: fusing the Gattneck and Gattblock basic modules with the attention fusion module respectively, adding the InteAttention sub-module to the Gattneck and Gattblock basic modules, and replacing the 3×3 operator of the Gattneck basic module with a 1×1 lightweight operator.

[0058] The method of the present invention will be further explained below with reference to specific applications:

[0059] like Figure 2 As shown, based on Figure 2 The schematic diagram shown illustrates how to obtain the motion features of a target person within a target scene, as detailed below:

[0060] First, image data of the target person in the target scene is acquired. The image data is then fed into a convolutional neural network for feature extraction. A candidate region network is added to the network, and the region of interest and the original image are simultaneously put into a pooling layer for feature extraction. This patent addresses the problems of excessive network depth and high computational requirements caused by using high-resolution representation. It optimizes the high-resolution neural network HRnet (High Resoultion Net) convolutional network to reduce computational cost and parameter count, thereby improving detection efficiency. During behavior recognition, 17 key points of the human body are detected. The 6 key points of the upper body are left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist. The 6 key points of the lower body are left hip, right hip, left knee, right knee, left ankle, and right ankle. The 5 key points of the head are left eye, right eye, left ear, right ear, and nose.

[0061] Meanwhile, targeted parameter studies were conducted for scenarios involving multiple people and occlusion to ensure the accuracy of skeleton feature extraction from the receptive field and to more accurately identify the behavior and actions of the target person.

[0062] The extraction model is lightweighted, including the following:

[0063] This lightweight human joint motion feature extraction model adds lightweight deep convolutional kernels to the preprocessing module and replaces the feature extraction part with a large number of parameters in the baseline algorithm SimpleBaseline. It also performs skip connections for channels with high computational cost and redesigns the scaling coefficients of the reduction and expansion layers of the inverted residual module. This solves the problem of redundant representation in multi-level feature fusion and improves the graph neural network GAT-net (Graph Attention Network) to ensure that the feature extraction capability of the feature map is not reduced while reducing the number of parameters and computational cost.

[0064] The specific improvement steps are as follows:

[0065] The multi-scale preprocessing submodule acquires and processes task feature information. This preprocessing module replaces the standard 3×3 convolution module with a lightweight 1x1 depthwise convolution and adds a 3x3 transposed convolution module. The transposed convolution is performed using an additive approach. The computational complexity P0 S for:

[0066] P S =(2H+2)(2W+2)αN(2K) 2 -1)

[0067] The parameter P of the inverse residual module is obtained by calculating formulas (2-1) and (2-2). IA The computational complexity G GA Where N is the number of operations, K is the number of key points, and α is the scaling factor.

[0068] The number of parameters P in the GAT-Net network GA =C in ×αC in +αC in ×3×3+αC in ×C out (2-1)

[0069] G GA = (2H+2)×W×C in ×αC in +(2H+2)×W×αC in ×

[0070] 3×3+H×W×αC in ×C out (2-2)

[0071] In equations (2-1) and (2-2), C in and C out Let H and W represent the input and output channels of the module, respectively. α is the scaling factor, and H and W represent the height and width of the feature map, respectively. The calculation formulas for the parameters of the InteAttention (intersperse attention) module, the attention mechanism module for scattered or disjointed key points, are shown in equation (2-3):

[0072] P IA =C in ×C mid +2×C mid ×C out (2-3)

[0073] Among them, C midThese are the compressed feature channels;

[0074] Based on the parameter P of the inverse residual module GA The computational complexity G GA An attention fusion module was introduced, and comparative experiments were conducted on the CrowdPose (CrowdPose Efficient Crowded Scenes Pose Estimation and ANew Benchmark) dataset, which is a key node class for skeletons, with different scaling factors. Low-dimensional and high-dimensional features were fused and aggregated proportionally to generate a spatial attention feature map.

[0075] The Gattneck and Gattblock modules both introduce dual attention modules to achieve cross-convolutional spatial information and accurate feature information acquisition. The Gattneck module replaces the 3×3 operator with a lightweight 1×1 operator, while the Gattblock module retains the basic framework of the Basicblock module. The Gattneck submodule can effectively reduce the amount of computation and some network parameters, while bringing certain performance improvements. Adding the InteAttention module can reduce the occurrence of redundant feature maps.

[0076] The formulas for calculating the number of parameters in the InteAttention module on the HRNet and Gatnet base modules are as follows:

[0077] P IA =C in ×C mid +2×C mid ×C out

[0078] P Gattneck =P IA +P CA (2-4)

[0079] P Gattblock =C in ×3×3+P IA +C out ×3×3+P CA (2-5)

[0080] Equations (2-4) and (2-5) are respectively the calculation formulas for the parameter quantities of the improved GAT-Net basic module;

[0081] The percentage reduction in parameters achieved by the Gattblock module compared to the Basicblock module is shown in formula (2-8):

[0082]

[0083]

[0084]

[0085] Equations (2-6) and (2-7) represent the percentage reduction in the number of parameters by the first and the remaining three Gattneck modules in Stage 1 (the improved GAT-Net), respectively. From the formulas, it can be seen that the scaling factor α in the inverted residual module is proportional to the percentage reduction in the number of parameters.

[0086] Based on the multi-scale image preprocessing module, Gattneck module, and Gattblock module obtained from the above steps, the Gattnet structure is selectively improved.

[0087] The first stage consists of two Gattblock modules that process the feature maps from the Stem module. For every halving of resolution, the number of feature channels is doubled, and a downsampling branch is implemented. The Gattblock module forms the foundation for the subsequent three stages.

[0088] By using non-contact sensors such as UWB radar to collect human vital signs information such as respiration and heart rate, as well as human location information, a human status database indexed by time is established. Ultimately, a system is formed that can use the above human information to compare with individual normal indicators and preset safety thresholds for the corresponding time period to assist in inferring individual health status, location, and behavior.

[0089] Based on the respiratory and heart rate conditions of different healthy and unhealthy individuals, the system monitors the normal and abnormal respiratory and heart rate information of individuals, constructs a time series of vital signs waveforms and converts them into images to form a dataset. Through the dataset, the system ultimately determines the normal or abnormal state of the current individual.

[0090] This invention can be applied to abnormal behavior monitoring systems to identify, monitor, and warn of abnormal human behavior, and classify the corresponding danger levels according to the degree of warning of abnormal behavior.

[0091] Example 2:

[0092] This invention also proposes a system 200 for identifying abnormal behavior based on multimodal information fusion, such as... Figure 3 As shown, it includes:

[0093] The initial unit 201 is used to acquire feature information of the target scene, establish a description sentence generation model based on the feature information of the target scene using a convolutional neural network, monitor the target scene through the description sentence generation model, acquire the spatiotemporal change information of the key points of the target person in the target scene, and monitor the target scene through a non-contact sensor to acquire the key point position information of the target person in the target scene.

[0094] The first recognition module 202 is used to establish an extraction model for extracting human behavior features based on the motion features of the target person, and to perform lightweight processing on the extraction model to generate a recognition model. The model is generated by the description statement to monitor the target scene, obtain the spatiotemporal change information of the key points of the person to be recognized, and send the spatiotemporal change information of the key points of the person to be recognized to the recognition model so that the recognition model can determine the human behavior features of the person to be recognized based on the spatiotemporal change information of the key points of the person to be recognized.

[0095] The second identification module 203 is used to establish a database for determining human physical characteristics based on the joint position information of the target person, monitor the target scene through a non-contact sensor, obtain the joint position information of the person to be identified, and send the joint position information to the database so that the database can determine the human physical characteristics of the person to be identified based on the joint position information of the person to be identified.

[0096] The fusion unit 204 is used to perform multimodal information fusion on the human behavior features and physical characteristics of the person to be identified in order to generate a dataset for determining the state of the person to be identified, and to determine whether the state of the person to be identified is abnormal based on the dataset.

[0097] The multimodal information fusion includes: establishing a waveform time series of the state of the person to be identified based on the human behavioral characteristics and physical signs of the person to be identified, converting the waveform time series into image data, and generating a dataset based on the image data.

[0098] The description statement generation model monitors the target scene and obtains spatiotemporal change information of key points of the target character within the target scene, including:

[0099] The target scene is monitored based on the description statement generation model to obtain image data of the target person in the target scene. The image data is then segmented to obtain the human foreground feature map of the target person. The region of interest image data in the human foreground feature map is determined. The image data and the region image data are then input into the pooling layer of the description statement generation model. The pooling layer detects multiple key points of the target person to extract the spatiotemporal change information of the target person's joints.

[0100] The extraction model includes various processing modules based on convolutional neural networks;

[0101] The processing module includes at least one of the following: a multi-scale image preprocessing module, an inverse residual module, an attention fusion module, a Gattneck basic module, a Gattblock basic module, and a Gattnet basic module.

[0102] The extraction model is lightweighted by modifying the structure of the multi-scale image preprocessing module, the Gattneck basic module, and the Gattblock basic module, including:

[0103] The modification to the structure of the multi-scale image preprocessing module includes: replacing the 3×3 convolution submodule of the multi-scale image preprocessing module with a lightweight 1×1 depth convolution submodule, and adding a 3×3 transposed convolution submodule to the multi-scale image processing module to perform transposed convolution by addition.

[0104] The structures of the Gattneck and Gattblock basic modules are modified, including: fusing the Gattneck and Gattblock basic modules with the attention fusion module respectively, adding the InteAttention sub-module to the Gattneck and Gattblock basic modules, and replacing the 3×3 operator of the Gattneck basic module with a 1×1 lightweight operator.

[0105] Example 3:

[0106] Based on the same inventive concept, this invention also provides a computer device, which includes a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement corresponding method flows or corresponding functions, thereby implementing the steps of the methods in the above embodiments.

[0107] Example 4:

[0108] Based on the same inventive concept, this invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of the method in the above embodiments.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0114] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for identifying abnormal behavior based on multimodal information fusion, characterized in that, The method includes: The system acquires feature information of the target scene, establishes a description sentence generation model based on the feature information of the target scene using a convolutional neural network, monitors the target scene using the description sentence generation model, acquires the spatiotemporal change information of the key points of the target person in the target scene, and monitors the target scene using a non-contact sensor to acquire the key point position information of the target person in the target scene. An extraction model for extracting human behavioral features is established based on the motion features of the target person, and the extraction model is lightweighted to generate a recognition model. The target scene is monitored through the model generated by the description statement to obtain the spatiotemporal change information of the key points of the person to be identified, and the spatiotemporal change information of the key points of the person to be identified is sent to the recognition model to enable the recognition model to determine the human behavioral features of the person to be identified based on the spatiotemporal change information of the key points of the person to be identified. A database for determining human physical characteristics is established based on the joint position information of the target person. The target scene is monitored by non-contact sensors to obtain the joint position information of the person to be identified and send the joint position information to the database so that the database can determine the human physical characteristics of the person to be identified based on the joint position information of the person to be identified. Multimodal information fusion is performed on the human behavioral features and physical characteristics of the person to be identified to generate a dataset for determining the state of the person to be identified, and the state of the person to be identified is determined based on the dataset. The multimodal information fusion includes: establishing a waveform time series of the state of the person to be identified based on the human behavioral characteristics and physical signs of the person to be identified, converting the waveform time series into image data, and generating a dataset based on the image data; The extraction model includes multiple processing modules based on convolutional neural networks; The processing module includes at least one of the following: a multi-scale image preprocessing module, an inverse residual module, an attention fusion module, a bottleneck Gattneck basic module, a residual Gattblock basic module, and a lightweight attention mechanism network Gattnet basic module. The lightweighting of the extraction model includes: modifying the structure of the multi-scale image preprocessing module, the Gattneck basic module, and the Gattblock basic module, including: The modification to the structure of the multi-scale image preprocessing module includes: replacing the 3×3 convolution submodule of the multi-scale image preprocessing module with a lightweight 1×1 depth convolution submodule, and adding a 3×3 transposed convolution submodule to the multi-scale image preprocessing module to perform transposed convolution by addition. The structures of the Gattneck and Gattblock basic modules are modified, including: fusing the Gattneck and Gattblock basic modules with the attention fusion module respectively, adding a distributed joint single attention mechanism InteAttention sub-module to the Gattneck and Gattblock basic modules, and replacing the 3×3 operator of the Gattneck basic module with a 1×1 lightweight operator.

2. The method according to claim 1, characterized in that, The model generates a description statement to monitor the target scene and obtain spatiotemporal change information of key points of the target person within the target scene, including: The target scene is monitored based on the description statement generation model to obtain image data of the target person in the target scene. The image data is then segmented to obtain the human foreground feature map of the target person. The region of interest image data in the human foreground feature map is determined. The image data and the region image data are then input into the pooling layer of the description statement generation model. The pooling layer detects multiple key points of the target person to extract the spatiotemporal change information of the target person's joints.

3. A system for identifying abnormal behavior based on multimodal information fusion, characterized in that, The system includes: The initial unit is used to acquire feature information of the target scene, establish a description sentence generation model based on the feature information of the target scene using a convolutional neural network, monitor the target scene through the description sentence generation model, acquire the spatiotemporal change information of the key points of the target person in the target scene, and monitor the target scene through a non-contact sensor to acquire the key point position information of the target person in the target scene. The first recognition module is used to establish an extraction model for extracting human behavior features based on the motion features of the target person, and to perform lightweight processing on the extraction model to generate a recognition model. The model is generated by the description statement to monitor the target scene, obtain the spatiotemporal change information of the key points of the person to be identified, and send the spatiotemporal change information of the key points of the person to be identified to the recognition model so that the recognition model can determine the human behavior features of the person to be identified based on the spatiotemporal change information of the key points of the person to be identified. The second identification module is used to establish a database for determining human physical characteristics based on the joint position information of the target person. It monitors the target scene through a non-contact sensor, obtains the joint position information of the person to be identified, and sends the joint position information to the database so that the database can determine the human physical characteristics of the person to be identified based on the joint position information of the person to be identified. The fusion unit is used to perform multimodal information fusion on the human behavioral features and physical characteristics of the person to be identified in order to generate a dataset for determining the state of the person to be identified, and to determine whether the state of the person to be identified is abnormal based on the dataset. The multimodal information fusion includes: establishing a waveform time series of the state of the person to be identified based on the human behavioral characteristics and physical signs of the person to be identified, converting the waveform time series into image data, and generating a dataset based on the image data; The extraction model includes multiple processing modules based on convolutional neural networks; The processing module includes at least one of the following: a multi-scale image preprocessing module, an inverse residual module, an attention fusion module, a Gattneck basic module, a Gattblock basic module, and a Gattnet basic module; The lightweighting of the extraction model includes: modifying the structure of the multi-scale image preprocessing module, the Gattneck basic module, and the Gattblock basic module, including: The modification to the structure of the multi-scale image preprocessing module includes: replacing the 3×3 convolution submodule of the multi-scale image preprocessing module with a lightweight 1×1 depth convolution submodule, and adding a 3×3 transposed convolution submodule to the multi-scale image preprocessing module to perform transposed convolution by addition. The structures of the Gattneck and Gattblock basic modules are modified, including: fusing the Gattneck and Gattblock basic modules with the attention fusion module respectively, adding the InteAttention sub-module to the Gattneck and Gattblock basic modules, and replacing the 3×3 operator of the Gattneck basic module with a 1×1 lightweight operator.

4. The system according to claim 3, characterized in that, The model generates a description statement to monitor the target scene and obtain spatiotemporal change information of key points of the target person within the target scene, including: The target scene is monitored based on the description statement generation model to obtain image data of the target person in the target scene. The image data is then segmented to obtain the human foreground feature map of the target person. The region of interest image data in the human foreground feature map is determined. The image data and the region image data are then input into the pooling layer of the description statement generation model. The pooling layer detects multiple key points of the target person to extract the spatiotemporal change information of the target person's joints.

5. A computer device, characterized in that, include: One or more processors; A processor is used to execute one or more programs; When the one or more programs are executed by the one or more processors, the method described in any one of claims 1-2 is implemented.

6. A computer-readable storage medium, characterized in that, It contains a computer program, which, when executed, implements the method as described in any one of claims 1-2.

Images