Abnormal behavior monitoring method and device, equipment, storage medium and program product
By using a behavior monitoring model with a lightweight deep convolution module, real-time analysis of surveillance video solves the problem of the inability to promptly detect sudden user events in unattended bank self-service areas. It enables rapid identification and alerts for abnormal user behavior and improves computational efficiency.
Patent Information
- Application Number
- CN202511126621.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-12
- Publication Date
- 2025-11-11
AI Technical Summary
When there is no one to assist in the self-service area of a bank, sudden events such as physical discomfort or falls may not be detected in time, leading to more serious injuries.
By using a behavior monitoring model based on a lightweight deep convolution module, image sequences in surveillance videos are analyzed in real time to extract temporal and spatial features, identify user behavior, and issue alerts.
It enables real-time monitoring of abnormal user behavior, especially the identification of falls, improving computing speed and allowing for timely measures to reduce harm.
Smart Images
Figure CN120932306A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video monitoring technology, and can be used in the financial technology field, particularly to an abnormal behavior monitoring method, device, electronic device, storage medium, and program product. Background Technology
[0002] Bank self-service areas are typically unattended. If a user experiences an emergency while using the equipment—such as falling due to illness, operational errors, or other unforeseen circumstances—the lack of detection can lead to more serious injuries. Therefore, a technology capable of real-time monitoring of abnormal user behavior in public service areas is needed to play a crucial role. Summary of the Invention
[0003] This application provides a method, apparatus, device, storage medium, and product for monitoring abnormal behavior, in order to solve the technical problem that existing technologies cannot monitor users' abnormal behavior.
[0004] Firstly, this application provides a method for monitoring abnormal behavior, the method comprising:
[0005] Based on the target area, acquire a sequence of images including the target object from surveillance video; the image sequence includes multiple consecutive frames.
[0006] Image sequences are input into a behavior monitoring model for real-time monitoring. The behavior monitoring model includes multiple lightweight deep convolutional modules. These modules are used to extract the temporal and spatial features of the images in the image sequence. The behavior monitoring model is used to determine the classification result of the target object's behavior based on the temporal and spatial features.
[0007] Determine whether the target object has fallen based on the classification results;
[0008] If it is determined that the target object has fallen, an alarm is issued and a target video is sent; the target video includes image frames of the target object falling.
[0009] In one embodiment, the lightweight depth convolution module includes a time-shifting unit;
[0010] The time-shifting unit is used to move channel information along the time dimension of the image, capture the temporal dynamics of the image, and obtain the temporal dimension features of the image.
[0011] In one embodiment, the time-shifting unit is used to move channel information along the time dimension of the image, capture the temporal dynamics of the image, and obtain the temporal dimension features of the image, specifically including:
[0012] The time-shifting unit divides the image channels into three parts: backward-shifting channels, forward-shifting channels, and fixed channels; it shifts the image channel information along the time dimension, passing the backward-shifting channel information to future frames and the forward-shifting channel information to past frames; it folds the shifted information into the channel dimension to obtain the temporal dimension features of the image, so that the feature map of each frame includes information from adjacent frames.
[0013] In one embodiment, the lightweight depth convolution module further includes a first standard convolution unit, a depth-separable convolution unit, an activation unit, a second standard convolution unit, and a normalization unit;
[0014] The first standard convolutional unit is used to obtain the spatial features of the image based on the output of the time-shifting unit;
[0015] Depthwise separable convolutional units are used to perform convolution operations on the input feature maps of each channel separately, and then the output results of each channel are mixed by 1*1 convolution.
[0016] The activation unit is used to perform non-linear activation on the output of the depthwise separable convolutional unit;
[0017] The second standard convolutional unit is used to perform standard convolution operation again on the output of the activation unit;
[0018] The normalization unit is used to normalize the output of the second standard convolutional unit to ensure that the number of channels in the input and output images of the lightweight depth convolutional module remains unchanged.
[0019] In one embodiment, the behavior monitoring model further includes a mapping module.
[0020] The mapping module is used to map the temporal and spatial features of an image sequence to behavior categories, thereby determining the classification result of the target object's behavior.
[0021] In one embodiment, before inputting the image sequence into the behavior monitoring model and determining the classification result of the target object's behavior, the method further includes:
[0022] The behavior monitoring model was pre-trained using a dataset that included sample videos of fall behaviors and normal behaviors of various target objects.
[0023] Optimize the trained behavior monitoring model using a low learning rate.
[0024] In one embodiment, the image sequence is input into the behavior monitoring model to determine the classification result of the target object's behavior, specifically including:
[0025] Based on image sequences, a behavior monitoring model is used to obtain the confidence level of the image sequences for each behavior category; the behavior categories include normal behavior and falling behavior.
[0026] In one embodiment, determining whether a target object has fallen based on the classification results specifically includes:
[0027] If the confidence level of the image sequence regarding the fall behavior exceeds a preset value, then it is determined that the target object has fallen.
[0028] If the confidence level of the image sequence regarding the fall behavior does not exceed the preset value, it is determined that the target object did not fall or further verification is required.
[0029] Secondly, this application provides an abnormal behavior monitoring device, comprising:
[0030] The acquisition module is used to acquire image sequences including target objects based on surveillance video of a target area; the image sequence includes multiple consecutive frame images;
[0031] The determination module is used to input the image sequence into the behavior monitoring model for real-time monitoring. The behavior monitoring model includes multiple lightweight deep convolution modules. The lightweight deep convolution modules are used to extract the temporal and spatial features of the images in the image sequence. The behavior monitoring model is used to determine the classification result of the target object's behavior based on the temporal and spatial features.
[0032] The determination module is also used to determine whether the target object has fallen based on the classification results; if it is determined that the target object has fallen, an alarm is issued and the target video is sent; the target video includes image frames of the target object falling.
[0033] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;
[0034] The memory stores the instructions that the computer executes;
[0035] The processor executes computer-executable instructions stored in memory to implement any of the methods described above.
[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods described above.
[0037] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.
[0038] This application provides an abnormal behavior monitoring method, which includes: acquiring an image sequence including a target object based on surveillance video of a target area; the image sequence including multiple consecutive frame images; inputting the image sequence into a behavior monitoring model for real-time monitoring, the behavior monitoring model including multiple lightweight deep convolutional modules, the lightweight deep convolutional modules being used to extract temporal and spatial features from the images in the image sequence, the behavior monitoring model being used to determine the classification result of the target object's behavior based on the temporal and spatial features; determining whether the target object has fallen based on the classification result; if it is determined that the target object has fallen, issuing an alarm and sending the target video; the target video including image frames of the target object falling. By inputting the image sequence into the behavior monitoring model for real-time monitoring, it is possible to determine whether the target object has fallen and take corresponding measures; secondly, the lightweight deep convolutional modules not only maintain the feature extraction effect but also improve the model's computational speed. Attached Figure Description
[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0040] Figure 1 A flowchart illustrating an abnormal behavior monitoring method provided in an embodiment of this application;
[0041] Figure 2 This is a schematic diagram of the structure of a lightweight depthwise convolution module provided in an embodiment of this application;
[0042] Figure 3 This is a schematic diagram of the operation process of a depth-separable convolutional unit in one embodiment of this application;
[0043] Figure 4 This is a schematic diagram of the time-shifting unit movement channel information provided in an embodiment of this application;
[0044] Figure 5 This is a schematic diagram of the structure of an abnormal behavior monitoring device provided in an embodiment of this application;
[0045] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.
[0046] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0048] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse.
[0049] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.
[0050] It should be noted that the abnormal behavior monitoring methods, devices, equipment, storage media and products provided in this application can be used in the field of video monitoring technology, the field of financial technology, and any field other than the field of video monitoring technology and the field of financial technology. The application fields of the abnormal behavior monitoring methods, devices, equipment, storage media and products in this application are not limited.
[0051] Bank self-service areas are typically unattended. If a user experiences an emergency while using the equipment—such as falling due to illness, operational errors, or other unforeseen circumstances—the lack of detection can lead to more serious injuries. Therefore, a technology capable of real-time monitoring of abnormal user behavior in public service areas is needed to play a crucial role.
[0052] Based on the aforementioned technical issues, the specific application scenarios of this application are servers, smart cameras, etc., which embed a software system with the behavior monitoring model described in this application to analyze the video from the camera and determine the user's abnormal behavior.
[0053] This application provides an abnormal behavior monitoring method that inputs image sequences into a behavior monitoring model for real-time monitoring to determine whether a target object has fallen, so as to take corresponding measures to deal with it. Secondly, the lightweight deep convolution module can not only maintain the effect of feature extraction, but also improve the computation speed of the model, aiming to solve the above-mentioned technical problems of the prior art.
[0054] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0055] like Figure 1 As shown, Figure 1 This is a flowchart illustrating an abnormal behavior monitoring method according to an embodiment of this application. The abnormal behavior monitoring method includes the following steps:
[0056] Step S101: Acquire an image sequence including the target object based on the surveillance video of the target area; the image sequence includes multiple consecutive frame images.
[0057] Specifically, cameras are installed in the target area to capture images from a top-down perspective, and the cameras output surveillance video of the target area. A target monitoring model is used to process the camera's surveillance video, monitoring target objects in the video in real time; models such as YOLO / SSD / Faster R-CNN are used. First, the real-time video is decomposed into single-frame images, and each frame is processed separately. For each detected target object, its position (bounding box coordinates) in the current frame is recorded. This position information can be stored in a data structure, such as a list or array, for subsequent processing. The image of the area containing the target object is extracted from the original frame image based on the recorded bounding box coordinates, ensuring that the extracted image contains the complete bounding box content.
[0058] The processed images, including those of the target object, are arranged chronologically, with each eight frames forming an image sequence. This sequence is then input into a behavior monitoring model to determine if the target object has exhibited a fall behavior. In other embodiments, the number of frames in the image sequence can be different; this embodiment only uses eight consecutive frames for illustration.
[0059] Step S102: Input the image sequence into the behavior monitoring model for real-time monitoring. The behavior monitoring model includes multiple lightweight deep convolution modules. The lightweight deep convolution modules are used to extract the temporal and spatial features of the images in the image sequence. The behavior monitoring model is used to determine the classification result of the target object's behavior based on the temporal and spatial features.
[0060] Specifically, the input image sequence to the behavior monitoring model is multiple, enabling online real-time monitoring. Multiple lightweight deep convolutional modules process the input images of the behavior monitoring model separately, which not only extracts image features but also reduces the computational load of model inference and improves the model's computational speed.
[0061] In one embodiment, step S102 specifically includes the following steps:
[0062] Based on image sequences, a behavior monitoring model is used to obtain the confidence level of the image sequences for each behavior category; the behavior categories include normal behavior and falling behavior.
[0063] Specifically, the classification results of the behavior monitoring model are used to indicate whether a fall occurs in an image sequence, as well as other possible related behaviors or states, such as falling behavior and normal behavior. Falling behavior refers to the target object in the video falling, losing balance, and falling to the ground; normal behavior refers to the target object in the video not falling or exhibiting other abnormal behavior and being in a normal activity state. The behavior monitoring model can also further subdivide falling behavior, such as minor falls, severe falls, and sudden illnesses.
[0064] Step S103: Determine whether the target object has fallen based on the classification results.
[0065] Specifically, the classification result output by the behavior monitoring model is a probability distribution used to characterize the confidence level of an image sequence belonging to each behavior type. For example, the classification result can be a vector, where each element corresponds to a behavior category, and the value is the confidence level of that category. For example, the output vector of the behavior monitoring model is: [0.1, 0.8, 0.05, 0.05], where the first element (0.1) represents the confidence level of normal behavior; the second element (0.8) represents the confidence level of falling behavior; the third element (0.05) represents the confidence level of a minor fall; and the fourth element (0.05) represents the confidence level of other abnormal behaviors. Based on the values in the output vector, it can be determined that the target object in the image sequence has fallen behavior.
[0066] In one embodiment, step S103 specifically includes the following steps:
[0067] If the confidence level of the image sequence regarding the falling behavior exceeds a preset value, then it is determined that the target object has fallen.
[0068] If the confidence level of the image sequence regarding the fall behavior does not exceed the preset value, it is determined that the target object did not fall or further verification is required.
[0069] Specifically, by setting preset values for the confidence levels of different behavior categories in the image sequence, more types of fall behaviors or abnormal behaviors can be distinguished, providing more detailed information for subsequent processing and facilitating rescue by staff or medical personnel.
[0070] Step S104: If it is determined that the target object has fallen, an alarm is issued and the target video is sent; the target video includes image frames of the target object falling.
[0071] This application uses image sequences to input into a behavior monitoring model for real-time monitoring to determine whether a target object has fallen, so as to take appropriate measures to deal with it; secondly, the lightweight deep convolution module can not only maintain the effect of feature extraction, but also improve the computation speed of the model.
[0072] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the lightweight deep convolution module provided in an embodiment of this application. The lightweight deep convolution module includes a time-shifting unit; the time-shifting unit is used to move channel information along the time dimension of the image to capture the temporal dynamics of the image and obtain the temporal dimension features of the image.
[0073] Specifically, in the process of processing image sequences, information in the temporal dimension is crucial because the motion and behavior changes of objects are reflected through time series. Traditional convolutional neural networks (such as ResNet) mainly focus on the extraction of spatial features, but have weak processing capabilities in the temporal dimension. The time-shifting module in this application integrates information in the temporal dimension into the convolution operation in a simple and efficient way, thereby enhancing the model's ability to perceive dynamic information.
[0074] In one embodiment, please refer to Figure 2 The lightweight depth convolution module also includes a first standard convolution unit, a depthwise separable convolution unit, an activation unit, a second standard convolution unit, and a normalization unit. The first standard convolution unit is used to obtain the spatial features of the image based on the output of the time-shifting unit. The depthwise separable convolution unit performs convolution operations on the input feature maps of each channel separately, and then performs 1*1 convolution on the output of each channel for channel mixing. The activation unit performs non-linear activation on the output of the depthwise separable convolution unit. The second standard convolution unit performs standard convolution operations again on the output of the activation unit. The normalization unit normalizes the output of the second standard convolution unit to ensure that the number of channels in the input and output images of the lightweight depth convolution module remains unchanged.
[0075] Specifically, the lightweight depthwise convolution module combines standard convolution and depthwise separable convolution structures. Compared to traditional convolution, it significantly reduces the number of model parameters and improves model computation speed while ensuring feature extraction performance. The output of the lightweight depthwise convolution module is as follows:
[0076] y=IN(F(x,{W i})+x)
[0077] Where x and y represent the input and output F(x, {W}) of the lightweight depthwise convolution module, respectively. i The module consists of standard convolutional units, depthwise separable convolutional units, ReLU activation units, time-shifting units, and normalization units. The number of channels in the input and output feature maps of the module remains unchanged. Figure 2 The example input / output channels are 32. For example... Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the computation process of a depthwise separable convolutional unit in one embodiment of this application. Compared to a single standard convolutional layer, the lightweight depthwise convolutional unit provided in this application reduces the number of parameters by a factor of four.
[0078] like Figure 3 As shown, depthwise separable convolution decomposes the traditional convolution operation into two independent steps: Depthwise Convolution: performs convolution operation on each input channel separately, without cross-channel convolution; Pointwise Convolution: uses a 1×1 convolution kernel to mix the channels of the output of the depthwise convolution. This decomposition can significantly reduce the number of parameters. Assuming the number of channels in the input feature map is C, the convolution kernel size is K×K, and the number of output channels is also (C): Traditional Convolution: The number of parameters is C×C×K×K. Depthwise Separable Convolution: The number of parameters is C×K×K+C×C. For the cases of K=3 and C=32, the number of parameters in traditional convolution is (32*32*3*3)=9216, while the number of parameters in depthwise separable convolution is 32×3×3+32×32=1792, a reduction of approximately 4 times.
[0079] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the movement of channel information by a time-shifting unit according to an embodiment of this application. The time-shifting unit is used to move channel information along the time dimension of an image, capturing the temporal dynamics of the image to obtain its temporal dimension features. Specifically, it includes:
[0080] The time-shifting unit divides the image channels into three parts: backward-shifting channels, forward-shifting channels, and fixed channels; it shifts the image channel information along the time dimension, passing the backward-shifting channel information to future frames and the forward-shifting channel information to past frames; it folds the shifted information into the channel dimension to obtain the temporal dimension features of the image, so that the feature map of each frame includes information from adjacent frames.
[0081] Specifically, determining future and past frames primarily relies on the temporal order of the video sequence. A video is a sequence of consecutive frames, each with a definite timestamp or sequential position. These timestamps or sequential positions are used to determine whether an image in the sequence is a "future frame" or a "past frame." Obtaining the timestamp of the current frame: Assuming the timestamp of the currently processed frame is t, future frames are those with timestamps greater than t; past frames are those with timestamps less than t. For example: Assuming the timestamps of the video frames are t1, t2, t3, ..., tn, and the timestamp of the current frame is tk: future frames are those with timestamps tk+1, tk+2, ..., tn; past frames are those with timestamps t1, t2, ..., tk-1. Alternatively, if the video frames do not have timestamps but have a definite order (such as a frame index), future and past frames can also be determined using the index. Get the index of the current frame: Assuming the index of the currently processed frame is i, determine the future frame: the future frame is the frame with an index greater than i; determine the past frame: the past frame is the frame with an index less than i.
[0082] The input feature map is given by the time-shift unit. Assume the shape of the input feature map is (N, T, C, H, W), where N is the batch size, T is the time step (number of frames), C is the number of channels, and H and W are the height and width of the feature map. Channel segmentation: The input feature map is divided into three parts: backward-shifting channels (the information of these channels will be passed to future frames); forward-shifting channels (the information of these channels will be passed to past frames); and fixed channels (the information of these channels remains unchanged). The backward-shifting channel information is passed to future frames. For example, for frame t, the channel information of frame t is moved to frame t+1; or the forward-shifting channel information is passed to past frames. For example, for frame t, the channel information of frame t is moved to frame t-1. For the first and last frames, since there are no corresponding past or future frames, they can be padded with zeros or have information from adjacent frames copied. The moved information is then folded into the channel dimension. The time-shift operation is only performed on a subset of channels, such as 8 / 1 channels. Figure 4As shown, the channel information of the image frame at time t=0 is passed to the image frame at time t=1, and the channel information at the original position is filled with 0 values; the channel information of the image frame at time t=n is then passed to the next image frame. In the time-shifting module of this application, time shifting is achieved based on frame indexing. By passing information from some channels to adjacent frames, dynamic information in the time dimension is captured, enhancing the abnormal behavior monitoring model's ability to perceive dynamic information.
[0083] In one embodiment, the behavior monitoring model further includes a mapping module, which maps the temporal and spatial features of the image sequence to behavior categories to determine the classification result of the target object's behavior.
[0084] Specifically, the output of the lightweight deep convolution module is pooled into a fixed-size feature vector, and the feature vector is mapped to the classification result of the behavior category to determine the behavior of the target object.
[0085] In one embodiment, before inputting the image sequence into the behavior monitoring model and determining the classification result of the target object's behavior, the following steps are also included:
[0086] The behavior monitoring model was pre-trained using a dataset that included sample videos of fall behaviors and normal behaviors of various target objects.
[0087] Optimize the trained behavior monitoring model using a low learning rate.
[0088] Specifically, the model is trained using a publicly available action recognition dataset, allowing it to fully learn the rich feature representations of various human actions. These features can be applied to specific action detection tasks such as fall behavior through transfer learning. Video clips of users falling in target areas or other scenarios are collected as training data. The fully connected layer used for action classification prediction in the pre-trained model is replaced with a binary fully connected layer that only predicts whether a fall has occurred. The model is then fine-tuned with a low learning rate to adapt to the new action monitoring task.
[0089] like Figure 5 As shown, Figure 5 A schematic diagram of an abnormal behavior monitoring device provided in an embodiment of this application. The abnormal behavior monitoring device 500 includes:
[0090] The acquisition module 501 is used to acquire an image sequence including the target object based on the surveillance video of the target area; the image sequence includes multiple consecutive frame images.
[0091] The determination module 502 is used to input the image sequence into the behavior monitoring model for real-time monitoring. The behavior monitoring model includes multiple lightweight deep convolution modules. The lightweight deep convolution modules are used to extract the temporal and spatial features of the images in the image sequence. The behavior monitoring model is used to determine the classification result of the target object's behavior based on the temporal and spatial features.
[0092] The determination module 502 is also used to determine whether the target object has fallen based on the classification result; if it is determined that the target object has fallen, an alarm is issued and the target video is sent; the target video includes image frames of the target object falling.
[0093] In one embodiment, the acquisition module 501 is further configured to pre-train the behavior monitoring model using a dataset; the dataset includes sample videos of fall behaviors and normal behaviors of various target objects;
[0094] Optimize the trained behavior monitoring model using a low learning rate.
[0095] In one embodiment, the determining module 502 is further configured to obtain the confidence level of the image sequence with respect to each behavior category based on the image sequence using a behavior monitoring model; the behavior categories include normal behavior and fall behavior.
[0096] In one embodiment, the determining module 502 is further configured to determine that the target object has fallen if the confidence of the image sequence regarding the fall behavior exceeds a preset value;
[0097] If the confidence level of the image sequence regarding the fall behavior does not exceed the preset value, it is determined that the target object did not fall or further verification is required.
[0098] The abnormal behavior monitoring device 500 provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0099] This application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;
[0100] The memory stores the instructions that the computer executes;
[0101] The processor executes computer-executable instructions stored in memory to implement any of the methods described above.
[0102] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6As shown, the electronic device 600 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the electronic device 600 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.
[0103] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.
[0104] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0105] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0106] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0107] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0108] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0109] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0110] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0111] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.
[0112] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0113] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.
[0114] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0115] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.
[0116] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0117] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A method for monitoring abnormal behavior, characterized in that, include: Based on surveillance video of the target area, an image sequence including the target object is acquired; the image sequence includes multiple consecutive frame images. The image sequence is input into a behavior monitoring model for real-time monitoring. The behavior monitoring model includes multiple lightweight deep convolutional modules. The lightweight deep convolutional modules are used to extract the temporal and spatial features of the images in the image sequence. The behavior monitoring model is used to determine the classification result of the target object's behavior based on the temporal and spatial features. Based on the classification results, determine whether the target object has fallen; If it is determined that the target object has fallen, an alarm is issued and a target video is sent; the target video includes image frames of the target object falling.
2. The method according to claim 1, characterized in that, The lightweight depthwise convolution module includes a time-shifting unit; The time-shifting unit is used to move channel information along the time dimension of the image, capture the temporal dynamics of the image, and obtain the temporal dimension features of the image.
3. The method according to claim 2, characterized in that, The time-shifting unit is used to move channel information along the time dimension of the image, capture the temporal dynamics of the image, and obtain the temporal dimension features of the image, specifically including: The time-shifting unit divides the image channels into three parts: a backward-moving channel, a forward-moving channel, and a fixed channel; it moves the image channel information along the time dimension, passing the backward-moving channel information to future frames and the forward-moving channel information to past frames; it folds the moved information to the channel dimension to obtain the time dimension features of the image, so that the feature map of each frame includes information from adjacent frame images.
4. The method according to claim 2, characterized in that, The lightweight depthwise convolution module further includes a first standard convolution unit, a depthwise separable convolution unit, an activation unit, a second standard convolution unit, and a normalization unit; The first standard convolutional unit is used to obtain the spatial features of the image based on the output of the time-shifting unit; The depthwise separable convolutional unit is used to perform convolution operations on the input feature maps of each channel, and then performs channel mixing on the output results of each channel using a 1*1 convolution. The activation unit is used to perform non-linear activation on the output of the depthwise separable convolution unit; The second standard convolutional unit is used to perform a standard convolution operation again on the output of the activation unit; The normalization unit is used to normalize the output of the second standard convolution unit to ensure that the number of channels in the input and output images of the lightweight depth convolution module remains unchanged.
5. The method according to claim 1, characterized in that, The behavior monitoring model also includes a mapping module. The mapping module is used to map the temporal and spatial features of the image sequence to the behavior category, thereby determining the classification result of the target object's behavior.
6. The method according to claim 1, characterized in that, Before inputting the image sequence into the behavior monitoring model to determine the classification result of the target object's behavior, the method further includes: The behavior monitoring model is pre-trained using a dataset that includes sample videos of fall behaviors and normal behaviors of various target objects. The trained behavior monitoring model is optimized using a low learning rate.
7. The method according to claim 1, characterized in that, The step of inputting the image sequence into the behavior monitoring model to determine the classification result of the target object's behavior specifically includes: Based on the image sequence, the confidence level of the image sequence with respect to each behavior category is obtained using the behavior monitoring model; the behavior categories include normal behavior and fall behavior.
8. The method according to claim 7, characterized in that, The step of determining whether the target object has fallen based on the classification result specifically includes: If the confidence level of the image sequence regarding the fall behavior exceeds a preset value, then it is determined that the target object has fallen. If the confidence level of the image sequence regarding the fall behavior does not exceed a preset value, it is determined that the target object has not fallen or requires further verification.
9. An abnormal behavior monitoring device, characterized in that, include: The acquisition module is used to acquire an image sequence including the target object based on the surveillance video of the target area; the image sequence includes multiple consecutive frame images; The determination module is used to input the image sequence into the behavior monitoring model for real-time monitoring. The behavior monitoring model includes multiple lightweight deep convolution modules. The lightweight deep convolution modules are used to extract the temporal and spatial features of the images in the image sequence. The behavior monitoring model is used to determine the classification result of the behavior of the target object based on the temporal and spatial features. The determining module is further configured to determine whether the target object has fallen based on the classification result; If it is determined that the target object has fallen, an alarm is issued and a target video is sent; the target video includes image frames of the target object falling.
10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.
12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.