Behavior classification methods, electronic devices, computer storage media

Abstract

This application discloses a behavior classification method, an electronic device, and a computer storage medium. The behavior classification method includes: acquiring a video stream and extracting multiple frames of images from the video stream in a temporal sequence; acquiring a densely populated area in each frame of images and extracting a first image feature of the densely populated area; acquiring an extended area of ​​the crowd in each frame of images based on the densely populated area and extracting a second image feature of the extended area; fusing the first image feature and the corresponding second image feature in a temporal sequence to obtain temporal features; and processing the temporal features based on a classification model to obtain a behavior classification. This approach improves the accuracy of behavior classification and enables precise location of a preset behavior occurrence area.

Description

Technical Field

[0001] This application relates to the field of video understanding technology, and in particular to a behavior classification method, electronic device, and computer storage medium. Background Technology

[0002] With the acceleration of urbanization and the advancement of smart cities, safety issues are receiving increasing attention. However, if the task of monitoring behavior is solely performed manually, it requires a large amount of human resources and is also susceptible to delays due to human error. Therefore, video-based automated behavior detection has become a popular research area; by identifying pre-defined behaviors through intelligent video analytics, safety incidents can be prevented in advance.

[0003] The inventors of this application discovered during their long-term research and development that in the prior art, traditional optical flow methods are typically used to analyze the speed and direction of crowd movement, and statistical analysis and logistic regression are used to analyze preset behaviors. However, these methods are mostly based on human experience and have great limitations. They cannot be applied to a large number of complex real-world scenarios and complex preset behaviors of various categories. Furthermore, traditional methods for detecting preset behaviors are difficult to detect because it is hard to detect the area where the preset behavior occurs, making it difficult to apply deep learning methods for video sequence classification. Summary of the Invention

[0004] The main technical problem addressed in this application is how to improve the accuracy of behavior classification and accurately locate the pre-defined behavior occurrence area.

[0005] To address the aforementioned technical problems, this application provides a behavior classification method. This method includes: acquiring a video stream and extracting multiple frames of images from the video stream in a temporal sequence; acquiring densely populated areas in each frame and extracting first image features from these densely populated areas; acquiring extended areas of the crowd in each frame and extracting second image features from these extended areas; fusing the first image features and the corresponding second image features in a temporal sequence to obtain temporal features; and processing the temporal features based on a classification model to obtain a behavior classification.

[0006] To address the aforementioned technical problems, this application provides an electronic device. This electronic device includes a memory and a processor, wherein the memory and processor are coupled; the memory stores program data, and the processor executes the program data to implement the aforementioned behavior classification method.

[0007] To address the aforementioned technical problems, this application provides a computer storage medium. This computer storage medium stores program instructions, which, when executed, implement the aforementioned behavior classification method.

[0008] The beneficial effects of this application are as follows: Unlike existing technologies, this application first determines the densely populated area and its first image features, the corresponding extended area and its second image features in multiple frames of a video stream. It then fuses the first and second image features and combines them with the temporal sequence of the multiple frames to form temporal features. Finally, it uses these temporal features to train a classification model to obtain behavior classification in the video stream. Therefore, this application can achieve behavior classification. Furthermore, because the temporal features used to train the classification model include the image features of the densely populated area and its extended area, it can accurately locate the predetermined behavior occurrence area, facilitating faster response to predetermined behaviors and improving security. Moreover, the temporal features of this application integrate the features of the predetermined behavior occurrence area and its extended area, making the classification model's network representation ability stronger and improving the accuracy of behavior classification. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0010] Figure 1 This is a flowchart illustrating an embodiment of the behavior classification method of this application;

[0011] Figure 2 yes Figure 1 A schematic diagram illustrating the specific process of step S12 in the behavior classification method of the embodiment;

[0012] Figure 3 yes Figure 1 A schematic diagram of the specific process of step S13 in the behavior classification method of the embodiment;

[0013] Figure 4 This is a flowchart illustrating an embodiment of the behavior classification method of this application;

[0014] Figure 5 This is a flowchart illustrating an embodiment of the behavior classification method of this application;

[0015] Figure 6 This is a schematic diagram of the structure of an embodiment of the transformer model of this application;

[0016] Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0017] Figure 8 This is a schematic diagram of the structure of an embodiment of the computer storage medium of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise expressly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0020] This application first proposes a behavior classification method, such as Figure 1 As shown, Figure 1 This is a flowchart illustrating an embodiment of the behavior classification method of this application. The behavior classification method of this embodiment specifically includes the following steps:

[0021] Step S11: Obtain the video stream and extract multiple frames of images from the video stream in time sequence.

[0022] To improve security monitoring, surveillance equipment, such as cameras, is typically installed. This equipment can acquire a top-down video stream of a person. In this embodiment, multiple frames can be extracted from the video stream in sequence, allowing the top-down video stream to more clearly demonstrate human behavior. Of course, in other embodiments, video streams from different directions can be acquired by using surveillance equipment positioned at different locations, depending on actual needs.

[0023] Specifically, in this embodiment, multiple frames of images are extracted from the video stream according to the time sequence with a first preset duration as the time interval; wherein, the time difference between the first frame and the last frame of the multiple frames is less than the time threshold, that is, the extracted multiple frames are within a certain time range, avoiding the extraction of multiple frames of images covering multiple behaviors with large time intervals (occurrence time).

[0024] This temporal sequence refers to the sequential order of images within the video stream, and it can be represented by numbers of varying sizes. Extracting multiple frames according to this temporal sequence ensures the accuracy of behavioral changes and improves the precision of behavior classification.

[0025] Step S12: Obtain the densely populated area in each frame image and extract the first image feature of the densely populated area.

[0026] Optionally, this embodiment can be achieved through, as follows: Figure 2 The method shown implements step S12. The method in this embodiment includes steps S21 and S22.

[0027] Step S21: Use a regression network model to process each frame of the image to obtain the first location information and the first size information of the densely populated area of ​​each frame of the image.

[0028] In this embodiment, the densely populated area is the preset area where the behavior occurs.

[0029] The regression network model is trained using each frame of the image to regress the densely populated area of ​​each frame, thereby obtaining the first location information and the first size information of the densely populated area of ​​each frame. The first location information is the location information of the center point of the densely populated area, and the first size information is the width and height of the densely populated area.

[0030] This embodiment uses a coordinate regression network to perform regression extraction on the preset behavior occurrence area, which helps to eliminate redundant information, facilitates the network to extract effective features, and can simultaneously detect multiple preset behaviors occurring at the same time.

[0031] In this embodiment, the regression network model is a coordinate regression network model, which is used to regress the center point position (x, y) and width and height dimensions (w, h) of the preset behavior occurrence area. The coordinate regression network model in this embodiment can contain 13 convolutional layers and 5 downsampling layers. After passing through a fully connected layer and being processed by a softmax function, the information (x, y, w, h) of the densely populated area is obtained.

[0032] Step S22: Use a feature extraction network model to process each densely populated area to obtain multiple first image features for each densely populated area.

[0033] The feature extraction network model is trained using each densely populated area (region image) to obtain multiple first image features for each densely populated area; these first image features are multidimensional features of the densely populated area.

[0034] The feature extraction network model in this embodiment can be a pruned version of the Inceptionv2 network model. This pruned Inceptionv2 network model replaces the sampling layers (i.e., pooling layers) under the convolutional layers of the traditional Inceptionv2 network model with max pooling layers, reduces the number of channels in the intermediate feature maps, and removes the batch normalization layer. This ensures the accuracy of the Inceptionv2 network model while reducing model processing time and improving the speed of behavior classification.

[0035] Step S13: Obtain the outward expansion region of the crowd in each frame image based on the densely populated area, and extract the second image features of the outward expansion region of the crowd.

[0036] Optionally, this embodiment can be achieved through, as follows: Figure 3 The method shown implements step S13. The method of this embodiment includes steps S31 to S33.

[0037] Step S31: Obtain the union region of the densely populated areas of the multi-frame images using the first location information and the first size information of the densely populated areas of the multi-frame images.

[0038] The union region of the densely populated areas of all frames is obtained by using the first location information and the first size information of the densely populated areas of all frames.

[0039] Specifically, the union region of the dense crowd areas of all extracted frame images is obtained by using the center point position (x, y) and width and height dimensions (w, h) of the dense crowd areas of all frame images; wherein, the union region is the merged region of the dense crowd areas of all frame images in the same coordinate system, that is, the union region covers the dense crowd areas of all frame images.

[0040] Step S32: Match the union region with each frame image to obtain the crowd expansion region of each frame image.

[0041] The center point position (x, y) and width and height dimensions (w, h) of the union region can be mapped to each frame of the image. The area determined by the center point position (x, y) and width and height dimensions (w, h) in each frame of the image is the crowd expansion area of ​​each frame of the image.

[0042] In other embodiments, other neural network models or deep learning methods can be used to obtain the densely populated areas and the outward expansion areas of the crowd in each frame of the image.

[0043] This embodiment does not limit the first image features of densely populated areas and the execution order of obtaining the outward expansion area of ​​the crowd in each frame of the image.

[0044] Step S33: Use a feature extraction network model to process the outer region of each population group to obtain multiple second image features of the outer region of each population group.

[0045] The feature extraction network model is trained using the outer region (region image) of each population to obtain multiple second image features of the outer region of each population; these second image features are multidimensional features of the outer region of the population.

[0046] In this embodiment, the same feature extraction network model as in step S22 can be used to extract features from the outer regions of each crowd. Using the same feature extraction network model to extract features from both the densely populated areas and the outer regions of the crowd can reduce feature extraction errors and improve the accuracy of behavior classification.

[0047] Step S14: Fuse the first image features and the corresponding second image features according to the time sequence to obtain the time sequence features.

[0048] As can be seen from the above analysis, images of different frames have different time sequence numbers. In this embodiment, the first image feature and the second image feature of the same frame image can be fused to obtain the fused features of each frame image. Then, the time sequence number corresponding to each frame image is combined to obtain the time sequence features of the behavior.

[0049] Step S15: Process the temporal features based on the classification model to obtain the behavior classification.

[0050] In this embodiment, the classification model is a transformer model. This embodiment uses temporal features to train the transformer model in order to output the corresponding behavior classification.

[0051] Unlike existing technologies, this embodiment first determines the densely populated area and its first image features, the corresponding extended area of ​​the crowd, and its second image features in multiple frames of the video stream. The first and second image features are then fused together, and combined with the temporal sequence of the multiple frames to form temporal features. Finally, the temporal features are used to train a classification model to obtain behavior classification in the video stream. Therefore, this embodiment can achieve behavior classification. Furthermore, because the temporal features used to train the classification model include the image features of the densely populated area and its extended area, it can accurately locate the preset behavior occurrence area, facilitating faster response to preset behaviors and improving safety. Moreover, the temporal features in this embodiment fuse the features of the preset behavior occurrence area and its extended area, making the classification model's network representation ability stronger and improving the accuracy of behavior classification.

[0052] This application further proposes another embodiment of the behavior classification method, such as Figure 4 As shown, the behavior classification method in this embodiment specifically includes the following steps:

[0053] Step S41: Obtain the video stream and extract multiple frames of images from the video stream in time sequence.

[0054] Step S41 is similar to step S11 above, and will not be repeated here.

[0055] Step S42: Process each frame of the image using a regression network model to obtain the first location information and the first size information of the densely populated area of ​​each frame of the image.

[0056] Step S42 is similar to step S21 above, and will not be repeated here.

[0057] Step S43: Determine whether each frame of the image contains a corresponding densely populated area; if yes, proceed to step S44; if no, proceed to step S41.

[0058] Step S44: Use a feature extraction network model to process each densely populated area to obtain multiple first image features for each densely populated area.

[0059] Before extracting multiple first image features of densely populated areas, it is first determined whether the extracted multiple frames of images all contain the corresponding densely populated areas. If not, it is considered that the multiple frames of images do not have the preset behavior and no further steps need to be performed; if so, it is considered that the preset behavior exists and further steps need to be performed to achieve behavior classification and localization of the preset behavior occurrence area.

[0060] Step S44 is similar to step S22 above, and will not be repeated here.

[0061] Step S45: Obtain the outward expansion region of the crowd in each frame image based on the densely populated area, and extract the second image features of the outward expansion region of the crowd.

[0062] Step S45 is similar to step S13 above, and will not be described in detail here.

[0063] Step S46: Fuse the first image features and the corresponding second image features according to the time sequence to obtain the time sequence features.

[0064] Step S46 is similar to step S14 above, and will not be described in detail here.

[0065] Step S47: Process the temporal features based on the classification model to obtain the behavior classification.

[0066] Step S47 is similar to step S15 above, and will not be described in detail here.

[0067] Compared to the above embodiments, this embodiment, before acquiring multiple first image features of a densely populated area, first determines whether all extracted multi-frame images contain the corresponding densely populated area. If one or more frames do not contain a densely populated area, it is considered that there is no preset behavior corresponding to those frames, and no further steps are needed. Images are then extracted from subsequent video streams in chronological order to observe subsequent behaviors. If all frames contain the corresponding densely populated area, it is considered that a preset behavior exists, and subsequent steps are needed to achieve behavior classification and location of the preset behavior's occurrence area. This method can improve the accuracy and efficiency of behavior classification.

[0068] This application further proposes another embodiment of the behavior classification method, such as Figure 5 As shown, the behavior classification method in this embodiment specifically includes the following steps:

[0069] Step S51: Obtain the video stream and extract multiple frames of images from the video stream in time sequence.

[0070] Step S51 is similar to step S41 above, and will not be repeated here.

[0071] Step S52: Scale each frame of image to its short side to the first preset size according to the original ratio.

[0072] Step S53: Collect multiple images of the second preset size on the scaled image as training samples for the regression network model, wherein the first preset size corresponds to the second preset size.

[0073] The first preset size is the side length of the training sample image, and the second preset size is the area of ​​the training sample image.

[0074] In this embodiment, the original images extracted from the video stream are segmented through steps S52 and S53 to form training samples of uniform size, which can improve the training speed and accuracy of the subsequent regression network model.

[0075] Step S54: Process each frame of the image using a regression network model to obtain the first location information and the first size information of the densely populated area of ​​each frame of the image.

[0076] Step S54 is similar to step S42 above, and will not be described in detail here.

[0077] Step S55: Determine whether each frame of the image contains a corresponding densely populated area; if yes, proceed to step S56; if no, proceed to step S51.

[0078] Step S56: Use a feature extraction network model to process each densely populated area to obtain multiple first image features for each densely populated area.

[0079] Step S56 is similar to step S44 above, and will not be repeated here.

[0080] Step S57: Obtain the outward expansion region of the crowd in each frame image based on the densely populated area, and extract the second image features of the outward expansion region of the crowd.

[0081] Step S57 is similar to step S45 above, and will not be repeated here.

[0082] Step S58: Fuse the first image features and the corresponding second image features according to the time sequence to obtain the time sequence features.

[0083] Step S58 is similar to step S46 above, and will not be described in detail here.

[0084] Step S59: Process the temporal features based on the classification model to obtain the behavior classification.

[0085] Step S59 is similar to step S47 above, and will not be described in detail here.

[0086] Compared to the above embodiments, this embodiment first segments the original images extracted from the video stream before using the regression network model to obtain densely populated areas, in order to form training samples of uniform size, which can improve the training speed and accuracy of the subsequent regression network model.

[0087] In one application scenario, a video stream of a crowd viewed from above is first captured, and 12 frames are extracted within a short time period (e.g., 5 seconds). Each frame is scaled down to 288 pixels on its shorter side while maintaining its original aspect ratio. Three 288*288 images are then uniformly sampled from this frame as training samples for a coordinate regression network model. Next, the coordinate regression network model is used to regress a preset behavioral region (dense crowd area). The coordinate regression network model is trained for each frame. If a regression region exists in the current frame and the previous 12 frames, the 12 frames are fed into the subsequent cropped Inceptionv2 network model; otherwise, no classification is required. The coordinate regression network model outputs the center position and width / height dimensions (x, y, w, h) of the dense crowd area. Then, the cropped Inceptionv2 network model is used to extract features from the dense crowd area of ​​each of the 12 frames, resulting in 12-dimensional features. The feature extraction network is the cropped Inceptionv2 network. The network is then used to calculate the union of all densely populated areas. The cropped Inceptionv2 network model is used to extract features from the crowd diffusion areas of the 12 frames of images, resulting in 12-dimensional features. Finally, the two 12-dimensional features are fused one by one into 12 features. The fused 12 features are then fed into the improved transformer model of this embodiment for behavior classification.

[0088] Among them, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of an embodiment of the transformer model of this application. The transformer model of this embodiment (not shown in the figure) includes an input layer 61, multiple encoding layers 62, and an output layer 63 connected in sequence. The input layer 61 is used to input temporal features, the encoding layers 62 are used to perform convolution processing on the temporal features, and the output layer 63 is used to obtain a behavior classification based on the processed result and output the classification of the behavior.

[0089] Furthermore, the encoding layer 62 of the transformer model in this embodiment includes a convolutional sub-layer 621, a normalization layer 622, and a sub-output layer (not shown) stacked sequentially. The convolutional sub-layer 621 is used to perform convolution processing on temporal features, the normalization layer 622 is used to normalize the result after convolution processing, and the sub-output layer is used to process the result after normalization processing to obtain and output the encoding result of the encoding layer 62. The sub-output layer includes a fully connected layer and a dropout layer.

[0090] The encoding result can be output to the next level encoding layer 62 or output layer 63.

[0091] The output layer 63 processes the following: pooling, softmax, and concatenation. Specifically: (Qi, Ki, Vi) = conv(1*1)(Xi), Headi = Softmax(pool(conv(Qi,Ki)))Vi, attention(Q, K, V) = Concat(head1, head2, head3, ...); where pool represents pooling, conv represents convolution, concat represents concatenation, Xi is the aforementioned temporal feature, i is the i-th channel of the split, concat is the concatenation process, and Q, K, V are the initialization matrices.

[0092] As can be seen from the above analysis, in the traditional transformer model, the initialization matrix Qi and the initialization matrix Ki are first multiplied, and then the result of the multiplication is processed by the Softmax function. However, the improved transformer model in this embodiment uses the convolution of the initialization matrix Qi and Ki instead of the matrix multiplication between them, which can reduce the computational cost of the transformer model.

[0093] Compared to the traditional transformer model, the transformer model in this embodiment uses fully connected layers and dropout layers instead of feed forward neural network layers, and does not require a decoding layer.

[0094] Compared to the traditional Transformer model, the Transformer model in this embodiment replaces Transformer matrix multiplication with a convolutional network, and uses fully connected layers and dropout layers instead of Feed Forward Neural Network layers. This reduces the computational cost and parameters of the Transformer model, reduces overfitting, improves its running speed, and ensures its classification accuracy, making it easier to deploy the Transformer model on mobile devices.

[0095] This application further proposes an electronic device, such as Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 80 of this embodiment includes a processor 81, a memory 82, an input / output device 83, and a bus 84.

[0096] The processor 81, memory 82, and input / output device 83 are respectively connected to bus 84. The memory 82 stores program data, and the processor 81 is used to execute the program data to implement the behavior classification method of the above embodiment.

[0097] In this embodiment, processor 81 can also be referred to as CPU (Central Processing Unit). Processor 81 may be an integrated circuit chip with signal processing capabilities. Processor 81 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 81 can be any conventional processor.

[0098] This application further proposes a computer storage medium, such as Figure 8 As shown, Figure 8 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 90 stores program instructions 91 thereon, which, when executed by a processor (not shown), implement the above-described behavior classification method.

[0099] In this embodiment, the computer storage medium 90 can be, but is not limited to, a USB flash drive, SD card, PD optical drive, portable hard drive, large-capacity floppy drive, flash memory, multimedia memory card, server, etc.

[0100] Unlike existing technologies, this application first identifies the densely populated area and its first image features, the corresponding extended area of ​​the crowd, and its second image features in multiple frames of a video stream. The first and second image features are then fused together, and combined with the temporal sequence of the multiple frames to form temporal features. Finally, the temporal features are used to train a classification model to obtain behavior classification in the video stream. Therefore, this application can achieve behavior classification. Furthermore, because the temporal features used to train the classification model include image features of the densely populated area and its extended area, it can accurately locate the predetermined behavior occurrence area, facilitating faster response to predetermined behaviors and improving public safety. Moreover, the temporal features of this application integrate the features of the predetermined behavior occurrence area and its extended area, making the classification model more expressive and improving the accuracy of behavior classification.

[0101] Furthermore, if the aforementioned functions are implemented as software functions and sold or used as independent products, they can be stored in a mobile terminal-readable storage medium. That is, this application also provides a storage device storing program data, which can be executed to implement the methods of the above embodiments. This storage device can be, for example, a USB flash drive, an optical disc, or a server. In other words, this application can be embodied in the form of a software product, which includes several instructions to cause a smart terminal to execute all or part of the steps of the methods described in the various embodiments.

[0102] In the description of this application, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0103] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0104] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0105] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (which may be a personal computer, server, network device, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0106] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A behavior classification method, characterized in that, include: Acquire a video stream and extract multiple frames of images from the video stream in a time sequence; Obtain densely populated areas in each frame of the image, and extract the first image features of the densely populated areas; Based on the densely populated area, the outward expansion region of the crowd in each frame of the image is obtained, and the second image feature of the outward expansion region of the crowd is extracted; The first image feature and the corresponding second image feature are fused according to the time sequence to obtain the time sequence feature; The temporal features are processed based on a classification model to obtain a behavior classification. The step of obtaining the crowd outward expansion region in each frame of the image includes: The union region of the densely populated areas in the multi-frame images is obtained using the first location information and the first size information of the densely populated areas in the multi-frame images; The union region is matched with each frame of the image to obtain the crowd expansion region of each frame of the image.

2. The behavior classification method according to claim 1, characterized in that, The step of obtaining densely populated areas in each frame of the image and extracting first image features of the densely populated areas includes: The image frame is processed using a regression network model to obtain the first location information and the first size information of the densely populated area in each frame of the image. Each of the densely populated areas is processed using a feature extraction network model to obtain multiple first image features for each of the densely populated areas.

3. The behavior classification method according to claim 2, characterized in that, The extraction of the second image features of the extended region of the crowd includes: The feature extraction network model is used to process each of the population expansion regions to obtain multiple second image features for each of the population expansion regions.

4. The behavior classification method according to claim 2, characterized in that, Further includes: Determine whether each frame of the image contains a corresponding densely populated area; If so, then the step of processing each of the densely populated areas using the feature extraction network model to obtain multiple first image features for each of the densely populated areas is performed; If not, then perform the steps of acquiring the video stream and extracting multiple frames of images from the video stream in a time sequence.

5. The behavior classification method according to claim 2, characterized in that, After acquiring the video stream and extracting multiple frames from the video stream in a time sequence, the method further includes: Each frame of the image is scaled to its short side to a first preset size according to its original proportions. Multiple images of a second preset size are captured from the scaled image and used as training samples for the regression network model; The first preset size corresponds to the second preset size.

6. The behavior classification method according to claim 1, characterized in that, The step of extracting multiple frames of images from the video stream in time sequence includes: Multiple frames of images are extracted from the video stream according to a preset time interval; Wherein, the time difference between the first frame and the last frame in the multi-frame image is less than a time threshold.

7. The behavior classification method according to claim 1, characterized in that, The classification model is a transformer model, which includes an input layer, multiple encoding layers, and an output layer connected in sequence. The input layer is used to input the temporal features, the encoding layers are used to perform convolution processing on the temporal features, and the output layer is used to obtain the behavior classification based on the processed results and output the behavior classification.

8. The behavior classification method according to claim 7, characterized in that, The encoding layer comprises a convolutional sub-layer, a normalization layer, and a sub-output layer stacked sequentially. The convolutional sub-layer is used to perform convolution processing on the temporal features. The normalization layer is used to normalize the result after the convolution processing. The sub-output layer is used to process the result after the normalization processing to obtain and output the encoding result of the encoding layer. The sub-output layer includes a fully connected layer and a dropout layer.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory is coupled to the processor; The memory is used to store program data, and the processor is used to execute the program data to implement the behavior classification method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that, It stores program instructions that, when executed, implement the behavior classification method according to any one of claims 1 to 8.

Images