Aggregation behavior detection method and device, equipment, medium and product

Abstract

The invention discloses an aggregation behavior detection method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a to-be-judged video frame and a preset target detection model, and the preset target detection model comprises a multi-scale mobile equipment visual converter model backbone network, a prediction network and a neck network comprising a preset target layer; performing target detection on the to-be-judged video frame according to a preset target detection model, and determining a target and target coordinate information included in the to-be-judged video frame; and determining whether an aggregation behavior exists or not according to the target coordinate information and a preset number threshold. And processing a to-be-judged video frame through the multi-scale backbone network, extracting features under different output sizes to carry out multi-scale target detection, and judging whether an aggregation behavior exists or not through obtained target coordinate information. According to the invention, accurate identification of the target is realized, accurate determination of the gathering behavior is ensured, timely taking of countermeasures in an emergency is facilitated, and safety and stability of the detected place are ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, device, equipment, medium and product for detecting aggregation behavior. Background Art

[0002] With the rapid progress of society, the flow of people in public places (such as bank branches or service halls, etc.) is increasing, and the safety and stability of their daily operations have attracted much attention. If an emergency occurs in a branch, such as when someone faints, people will enthusiastically gather around to offer help, or when there is a fight, people will gather together out of curiosity to check the situation. If this is not handled in time, it may not only affect the normal operation of the branch, but also bring potential safety hazards.

[0003] Video surveillance in public places is an important part of ensuring safety. It can grasp the security dynamics in public places in real time through all-round monitoring, and can also provide strong evidence for accident investigation afterwards. By performing target detection on the video surveillance screen through the detection model, it can identify the people in the surveillance screen and achieve the purpose of monitoring the gathering situation.

[0004] However, surveillance equipment in public places generally adopts an overhead shooting method. Although this method can cover a larger area, it often causes human targets to appear smaller in the picture and are easily blocked by other objects or people, thus affecting the monitoring effect. In addition, there are many detection model parameters, making it difficult to deploy on mobile devices with limited computing power. Summary of the invention

[0005] The present invention provides a method, device, equipment, medium and product for detecting gathering behavior, so as to realize the detection of gathering behavior.

[0006] According to one aspect of the present invention, a method for detecting aggregation behavior is provided, comprising:

[0007] Obtain a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer MobileViTv3 model backbone network, a prediction network, and a neck network including a preset target layer;

[0008] Performing target detection on the to-be-judged video frame according to the preset target detection model to determine the target and target coordinate information included in the to-be-judged video frame;

[0009] Determine whether there is aggregation behavior based on the target coordinate information and a preset quantity threshold.

[0010] According to a second aspect of the present invention, there is provided a gathering behavior detection device, comprising:

[0011] A video frame acquisition module is used to acquire a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer MobileViTv3 model backbone network, a prediction network, and a neck network containing a preset target layer;

[0012] A target detection module, used to perform target detection on the video frame to be judged according to the preset target detection model, and determine the target and target coordinate information included in the video frame to be judged;

[0013] The aggregation behavior determination module is used to determine whether there is aggregation behavior based on the target coordinate information and a preset quantity threshold.

[0014] According to a third aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the aggregation behavior detection method described in any embodiment of the present invention.

[0018] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the aggregation behavior detection method described in any embodiment of the present invention when executed.

[0019] According to a fifth aspect of the present invention, an embodiment of the present invention further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the aggregation behavior detection method of any embodiment of the present invention is implemented.

[0020] The technical solution of the embodiment of the present invention obtains a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer model backbone network, a prediction network, and a neck network including a preset target layer; performs target detection on the video frame to be judged according to the preset target detection model, and determines the target and target coordinate information included in the video frame to be judged; determines whether there is aggregation behavior according to the coordinate information of each target and a preset quantity threshold. The video frame to be judged is processed by including a multi-scale model backbone network, and features at different output sizes are extracted to perform multi-scale target detection, and whether there is aggregation behavior is determined by the obtained target coordinate information. Accurate identification of targets is achieved, and accurate determination of aggregation behavior is guaranteed, which helps to take timely response measures in emergency situations to ensure the safety and stability of the detected location.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a method for detecting aggregation behavior provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of a method for detecting aggregation behavior provided according to Embodiment 2 of the present invention;

[0025] Figure 3 is a diagram showing an example of a model structure of a method for detecting agglomeration behavior according to Embodiment 2 of the present invention;

[0026] Figure 4 is a structural schematic diagram of a gathering behavior detection device provided according to Embodiment 3 of the present invention;

[0027] Figure 5 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 A flowchart of a gathering behavior detection method is provided for the first embodiment of the present invention. This embodiment is applicable to the detection of gathering behavior. The method can be performed by a gathering behavior detection device. The gathering behavior detection device can be implemented in the form of hardware and / or software. The gathering behavior detection device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110, obtaining a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer MobileViTv3 model backbone network, a prediction network, and a neck network including a preset target layer.

[0033] In this embodiment, the video frame to be determined can be understood as each frame of an image in a video used for performing aggregation behavior detection.

[0034] The preset target detection model can be understood as a model for identifying targets contained in an image, such as an improved YOLOv5 model. The backbone network of the multi-scale mobile device visual transformer MobileViTv3 model can be understood as a pig liver network for feature extraction. The prediction network can be understood as a network for target prediction. The neck network can be understood as a network for feature fusion through a preset target layer, and the preset target layer can be understood as a layer for feature fusion according to different image output scales.

[0035] Specifically, the processor can obtain the video frames to be judged in the area where aggregation behavior detection is required, which are captured by the relevant shooting equipment, and the preset target detection model for target detection which is pre-stored in the associated storage medium.

[0036] S120, performing target detection on the video frame to be judged according to a preset target detection model, and determining the target and target coordinate information included in the video frame to be judged.

[0037] In this embodiment, the target can be understood as the target subject of the aggregation behavior judgment. The target coordinate information can be understood as representing each target position in the form of coordinates.

[0038] Specifically, the processor can input the video frame to be judged into a preset target detection model, extract the features of the video frame to be judged through the multi-scale mobile device MobileViTv3 model backbone network of the preset target detection model, and input the extracted features at different scales into the neck network, perform feature fusion through the preset target layer in the neck network, obtain fused multi-scale features, and then input the fused multi-scale features into the multi-scale prediction network, perform multi-scale prediction through the prediction network, determine the position and confidence of targets of different scales included in the video frame to be judged, and then determine the target and target coordinate information.

[0039] S130. Determine whether there is aggregation behavior based on the coordinate information of each target and a preset quantity threshold.

[0040] In this embodiment, the preset quantity threshold can be understood as a target quantity threshold set for determining whether the aggregation condition is met.

[0041] Specifically, the processor can determine the distance between any two targets based on the coordinate information of each target, and then determine the number of other targets within a certain radius relative to each target, and then compare the maximum number of other targets with a preset number threshold. If it is greater, it is determined that there is a clustering behavior, otherwise there is no clustering behavior.

[0042] The technical solution of the embodiment of the present invention obtains a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer model backbone network, a prediction network, and a neck network including a preset target layer; performs target detection on the video frame to be judged according to the preset target detection model, and determines the target and target coordinate information included in the video frame to be judged; determines whether there is aggregation behavior according to the coordinate information of each target and a preset quantity threshold. The video frame to be judged is processed by including a multi-scale model backbone network, and features at different output sizes are extracted to perform multi-scale target detection, and whether there is aggregation behavior is determined by the obtained target coordinate information. Accurate identification of targets is achieved, and accurate determination of aggregation behavior is guaranteed, which helps to take timely response measures in emergency situations to ensure the safety and stability of the detected location.

[0043] As a first optional embodiment of the first embodiment, after determining whether there is a gathering behavior according to the coordinate information of each target and the preset quantity threshold, the method further includes:

[0044] When aggregation behavior occurs, an alarm message is generated and the associated objects are notified.

[0045] In this embodiment, the warning information can be understood as relevant information for notifying the current existence of gathering behavior, such as the number and location of the gathering persons, etc. The associated object can be understood as an object that processes the distance behavior, such as a management personnel, etc.

[0046] Specifically, when the processor determines that there is a gathering behavior, it can determine the location where the gathering behavior occurs and the number of gathering targets through the target coordinate information, generate an alarm message and send it to the associated object to notify the associated object to handle the gathering behavior. The processor can also record the determined number of targets and analyze the customer flow data of bank outlets, providing banks with valuable business analysis basis. Through the analysis of the data, banks can understand the customer flow distribution of outlets, peak hours and other information, providing strong support for optimizing service processes and improving customer satisfaction.

[0047] In the first optional embodiment of the first embodiment of the present invention, through such a setting, when it is determined that there is a gathering behavior, an alarm message is generated immediately and the related objects are notified in time to ensure that response measures are taken in time in an emergency.

[0048] Embodiment 2

[0049] Figure 2 This is a flow chart of a method for detecting agglomeration behavior provided in Embodiment 2 of the present invention. This embodiment is a further refinement of the above embodiment. Figure 2 As shown, the method includes:

[0050] S201, obtaining a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer MobileViTv3 model backbone network, a prediction network, and a neck network including a preset target layer.

[0051] S202, extracting features of the video frame to be judged according to the backbone network of the multi-scale MobileViTv3 model, and determining feature maps at different output scales.

[0052] Specifically, the processor can input the video frame to be judged into the multi-scale MobileViTv3 model backbone network, perform multi-scale feature extraction, and obtain feature maps at different output scales.

[0053] Furthermore, the backbone network of the multi-scale MobileViTv3 model includes a MobileViTv3 model, a fast spatial pyramid pooling module, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer. Accordingly, the steps of extracting features of the video frame to be judged according to the backbone network of the multi-scale MobileViTv3 model and determining feature maps at different output scales can be refined as follows:

[0054] The video frame to be judged is processed according to the first convolutional layer, the second convolutional layer and the MobileViTv3 model to obtain a first feature map at a first output scale; the first feature map is processed according to the third convolutional layer and the MobileViTv3 model to obtain a second feature map at a second output scale; the second feature map is processed according to the fourth convolutional layer and the MobileViTv3 model to obtain a third feature map at a third output scale; the third feature map is processed according to the fifth convolutional layer, the MobileViTv3 model and the fast spatial pyramid pooling module to obtain a fourth feature map at a fourth output scale.

[0055] In this embodiment, the MobileViTv3 model is a third-generation model for feature extraction. Compared with the initial version, the model replaces the 3×3 convolution layer with a 1×1 convolution layer, and fuses the features of the local representation block and the global representation block. At the same time, before generating the MobileViT Block output, the input features are added to the fusion block as the last step, and the ordinary 3×3 convolution layer in the local representation block is replaced with a deep 3×3 convolution layer. This model has both the superior performance of the ViT model for accurate detection and the lightweight advantages of the CNN model. It can greatly reduce the model parameters, is mobile-friendly, and has the possibility of being deployed on mobile devices.

[0056] In the present embodiment, the Fast Spatial Pyramid Pooling Module (SPPF) can be understood as an algorithm module that quickly extracts and fuses multi-scale feature information. The first convolutional layer can be understood as a convolutional layer under the first parameter configuration, for example, the parameters of the convolutional layer can be expressed as 0-p1 / 2, the number of layers is 0, p1 indicates that the number of convolutional layers is 1, and the downsampling rate is 2, and the downsampling rate indicates the multiple of image reduction, for example, if the original size of the input image is 640*640, then after the downsampling rate is 2, it is reduced to 320*320. The second convolutional layer can be understood as a convolutional layer under the second parameter configuration, for example, the parameters of the convolutional layer can be expressed as 1-p2 / 4, the number of layers is 1, p1 indicates that the number of convolutional layers is 2, and the downsampling rate is 4. The third convolutional layer can be understood as the convolutional layer under the third parameter configuration. For example, the parameters of the convolutional layer can be expressed as 3-p3 / 8, the number of layers is 3, p3 means the number of convolutional layers is 3, the downsampling rate is 8, and the multiple of image reduction. For example, if the original size of the input image is 640*640, then after the downsampling rate is 8, it is reduced to 80*80. The fourth convolutional layer can be understood as the convolutional layer under the fourth parameter configuration. For example, the parameters of the convolutional layer can be expressed as 5-p4 / 16, the number of layers is 5, p4 means the number of convolutional layers is 4, and the downsampling rate is 16. The fifth convolutional layer can be understood as the convolutional layer under the fifth parameter configuration. For example, the parameters of the convolutional layer can be expressed as 7-p5 / 32, the number of layers is 7, p5 means the number of convolutional layers is 5, and the downsampling rate is 32. Among them, the stride of the convolutional layer in the backbone network is 2, and the padding is 1. The output scale can be understood as the processed image size or resolution, for example, it can include 20×20, 40×40, 80×80 and 160×160. The feature map can be understood as reflecting the important information contained in the video frame.

[0057] Specifically, the processor can input the video frame to be judged into the backbone network of the multi-scale MobileViTv3 model, first process the video frame to be judged through the first convolution layer, and input the processed result into the second convolution layer for processing, and input the processed result into the MobileViTv3 model for processing. Through the MobileViTv3 model, the input image is first connected to 3×3 standard convolution and downsampled by 2 times, and then passed through multiple MV2 modules, wherein the MV2 module with a step size of 1 performs feature extraction, and the MV2 module with a step size of 2 performs 2 times downsampling, and then the obtained feature map is passed into the MobileViTv3 Block module and the MV2 module with a step size of 2, and then the 3×3 standard convolution is used for channel compression, and finally the global average pooling is performed to obtain the first feature map at the first output scale (for example, the first output scale can be 160×160). The first feature map can be processed by the third convolution layer, and the processed result is input into the MobileViTv3 model to obtain the second feature map at the second output scale (for example, the second output scale can be 80×80). The second feature map can be processed by the fourth convolution layer, and the output result is input into the MobileViTv3 model to obtain a third feature map at a third output scale (for example, 40×40). The third feature map can be processed by the fifth convolution layer, and the processed result is input into the MobileViTv3 model, and the model output result is input into the fast spatial pyramid pooling module for processing to obtain a fourth feature map at a fourth output scale (for example, 20×20).

[0058] For example, in order to facilitate understanding of the structure of the preset target detection model in the present invention, a specific example can be used for illustration. Figure 3 This is an example diagram of a model structure of a method for detecting agglomeration behavior provided in the second embodiment of the present invention. Figure 3As shown, the overall model can be an improved YOLOv5 model. The leftmost side of the figure is the backbone network of the multi-scale MobileViTv3 model, which replaces the original backbone feature extraction network with a more efficient and lightweight MobileViTv3 network. The video frame to be judged (the size of the video frame is 640*640*3, where 640*640 is the feature map size and 3 is the depth) is input from the top, and the first feature map under the first output scale (output size is 160*160*128) is obtained after being processed by the first convolutional layer (output size is 320*320*64), the second convolutional layer (output size is 160*160*128) and the MobileViTv3 model; the first feature map is processed by the third convolutional layer (output size is 80*80*256) and the MobileViTv3 model. The second feature map at the second output scale (output size is 80*80*256) is obtained; the second feature map is processed by the fourth convolutional layer (output size is 40*40*512) and the MobileViTv3 model to obtain the third feature map at the third output scale (output size is 40*40*512); the third feature map is processed by the fifth convolutional layer (output size is 20*20*1024), the MobileViTv3 model and the SPPF module to obtain the fourth feature map at the fourth output scale (output size is 20*20*1024).

[0059] S203, fusing the feature maps according to the neck network to obtain fused feature maps at each output scale.

[0060] In this embodiment, the fused feature map can be understood as the result of the fusion of shallow features and deep features.

[0061] Specifically, the feature maps at different output scales output by the backbone network of the multi-scale MobileViTv3 model can be fused through the multi-scale preset target layer in the neck network to obtain the fused feature map at the corresponding output scale.

[0062] Furthermore, the neck network includes a first preset target layer, a second preset target layer and an initial target layer. Accordingly, the steps of fusing the feature maps according to the neck network to obtain the fused feature maps at each output scale can be refined as follows:

[0063] The fourth feature map at the fourth output scale is processed by the initial target layer to obtain a first fused feature map at the fourth output scale; the fourth feature map is upsampled by the initial target layer to obtain a fifth feature map and combined with the third feature map at the third output scale to determine the second fused feature map at the third output scale; the third feature map is upsampled by the first preset target layer to obtain a sixth feature map and combined with the second feature map at the second output scale to determine the third fused feature map at the second output scale; the second feature map is upsampled by the second preset target layer to obtain a seventh feature map and combined with the first feature map at the first output scale to determine the fourth fused map at the first output scale.

[0064] In this embodiment, the initial target layer can be understood as a layer for processing a feature map at the fourth output scale. The first preset target layer and the second preset target layer can have the same structure. The fifth feature map can be understood as a feature map at the third output scale obtained after upsampling. The sixth feature map can be understood as a feature map at the second output scale after upsampling. The seventh feature map can be understood as a feature map at the first output feature after upsampling.

[0065] Specifically, the fourth feature map at the fourth output scale can be convolved through the initial target layer, and the result of downsampling by the first preset target layer is fused with the fourth feature map after convolution to obtain a first fused feature map at the fourth output scale; the fourth feature map is upsampled through the initial target layer to obtain a fifth feature map, which is added to the third feature map at the third output scale to obtain the addition result at the third output scale, and is fused with the result of downsampling by the second preset target layer to obtain a second fused feature map; the third feature map is upsampled through the first preset target layer to obtain a sixth feature map, which is added to the second feature map at the second output scale, and the result is fused with the result of downsampling by the first preset target layer to obtain a third fused feature map at the second output scale; the second feature map is upsampled through the second preset target layer to obtain a seventh feature map, which is fused with the first feature map at the first output scale to obtain a fourth fused feature map at the first output scale.

[0066] For example, Figure 3As shown, the middle box corresponds to the neck network. It can be seen from the figure that the neck network includes three target layers, which are the initial target layer, the first preset target layer and the second preset target layer from bottom to top. Among them, in each target layer, Conv represents the convolution layer, upsample represents the upsampling layer, concat represents the concatenation layer, and the C3_1 module represents the C3 convolutional neural network module, that is, a specific convolutional neural network module. The fourth feature map (represented by T4 in the figure) is upsampled and output through the upsample layer of the initial target layer to obtain the fifth feature map (represented by T5 in the figure) at the third output scale, and is added to the third feature map (represented by T3 in the figure) to obtain the addition result (T3+T5) at the third output scale and input to the first preset target layer, and the sixth feature map (represented by T6 in the figure) is upsampled through the first preset target layer and added to the second feature map (represented by T2 in the figure) to obtain the addition result (T2+T6); the addition result (T2+T6) is upsampled through the second preset target layer to obtain The seventh feature map (represented by T7 in the figure) is added to the first feature map (represented by T1 in the figure) at the first output scale and processed through the C3_1 module to obtain the fourth fused feature map at the first output scale. The second preset target layer downsamples the fourth fused feature map, and transmits the downsampling result to the C3_1 module on the right side of the first preset target layer for processing to obtain the third fused feature map. The third fused feature map is processed by the C3_1 module on the right side of the initial target layer to obtain the second fused feature map. The second fused feature map is downsampled by the initial target layer and combined with the fourth feature map after convolution to obtain the first fused feature map. The rightmost box is the prediction network. It can be seen that each output fused feature map corresponds to a prediction network that performs prediction at the output level.

[0067] S204, performing target prediction on the fused feature map based on the prediction network and the soft non-maximum suppression Soft-NMS algorithm, and determining the target and target coordinate information included in the video frame to be judged.

[0068] Specifically, the processor can perform target prediction at different levels on the fused feature map at each level based on a multi-scale prediction network, so as to determine detection frames of different sizes, and screen the overlapping detection frames through the soft non-maximum suppression Soft-NMS algorithm to determine the target and target coordinate information included in the video frame to be judged. Among them, the Soft-NMS algorithm will set a penalty mechanism. When the score of the overlapping detection frame is greater than the threshold, it is not directly set to zero, but the scores of these prediction frames are reduced. Even if the scores are high, they may be retained. This algorithm can effectively reduce the missed detection rate and improve the accuracy of target detection. The function of the Soft-NMS algorithm is shown as follows:

[0069]

[0070] Where represents the i-th detection box, represents the score of the i-th detection box, represents the NMS threshold, and M represents the detection box with the highest score.

[0071] S205: Determine the distance between the targets based on the coordinate information of the targets.

[0072] In this embodiment, the distance value can be understood as a quantity representing the distance between each target.

[0073] Specifically, the processor may traverse each detected target in turn, and calculate the Euclidean distance between each target and other targets with the target as the center, and the calculation formula is as follows:

[0074]

[0075] Among them, the target coordinate information is (x, y).

[0076] S206. Determine the number of targets that meet the aggregation condition according to each distance value and a preset threshold radius.

[0077] In this embodiment, the preset threshold radius can be understood as a radius for determining the correlation between targets, for example, the closer the distance, the more likely there is a clustering behavior. The clustering condition can be understood as a condition for determining whether there is a clustering behavior. The number of targets can be understood as the number of clustered targets.

[0078] Specifically, the processor may take each target as the central target in turn, compare the distance values ​​of other targets relative to the central target with a preset threshold radius, count the targets that are smaller than the preset threshold radius, and determine the number of targets that meet the distance condition.

[0079] S207: If the target quantity is greater than a preset quantity threshold, it is determined that aggregation behavior exists.

[0080] Specifically, the processor may compare the target quantity with a preset quantity threshold, and if the target quantity is greater than the preset quantity threshold, it is determined that aggregation behavior exists.

[0081] S208. Otherwise, determine that there is no aggregation behavior.

[0082] Specifically, if the number is less than or equal to a preset quantity threshold, it is determined that there is no aggregation behavior.

[0083] The technical solution of the embodiment of the present invention is to extract features from the video frame to be judged through the multi-scale MobileViTv3 model backbone network, extract the features of the target in the monitoring image by using the inverse residual structure of the linear bottleneck and the deep separable convolution, obtain the multi-scale semantic features of the image and enhance the correlation between the features through the attention mechanism SE-Net, so as to reduce the number of parameters of the model while improving the detection accuracy. The feature maps at different output scales are fused through the neck network to obtain the fused feature maps at each output scale, so as to achieve the purpose of detecting smaller targets, which can effectively reduce the missed detection rate and improve the target detection accuracy. The fused feature map is used to predict the target through the prediction network and the soft non-maximum suppression algorithm, and the target and target coordinate information included in the video frame to be judged are determined, and the distance value between any two targets is determined by the target coordinate information. The distance value is combined with the preset threshold radius and the preset number threshold to determine whether there is a clustering behavior. It is convenient to quickly identify and respond to these events, take effective measures to deal with them, and reduce losses and risks. It helps banks to take timely countermeasures in emergency situations to ensure the safety and stability of the detected locations.

[0084] Embodiment 3

[0085] Figure 4 This is a schematic diagram of the structure of a gathering behavior detection device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0086] The video frame acquisition module 41 is used to acquire the video frame to be judged and the preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer MobileViTv3 model backbone network, a prediction network and a neck network including a preset target layer;

[0087] The target detection module 42 is used to perform target detection on the video frame to be judged according to the preset target detection model, and determine the target and target coordinate information included in the video frame to be judged;

[0088] The gathering behavior determination module 43 is used to determine whether there is a gathering behavior based on the target coordinate information and a preset quantity threshold.

[0089] The technical solution of the embodiment of the present invention obtains a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer model backbone network, a prediction network, and a neck network including a preset target layer; performs target detection on the video frame to be judged according to the preset target detection model, and determines the target and target coordinate information included in the video frame to be judged; determines whether there is aggregation behavior according to the coordinate information of each target and a preset quantity threshold. The video frame to be judged is processed by including a multi-scale model backbone network, and features at different output sizes are extracted to perform multi-scale target detection, and whether there is aggregation behavior is determined by the obtained target coordinate information. Accurate identification of targets is achieved, and accurate determination of aggregation behavior is guaranteed, which helps to take timely response measures in emergency situations to ensure the safety and stability of the detected location.

[0090] Furthermore, the target detection module 42 includes:

[0091] A first determination unit is used to extract features of the video frame to be determined according to the multi-scale MobileViTv3 model backbone network, and determine feature maps at different output scales;

[0092] A second determining unit is used to perform fusion processing on each of the feature maps according to the neck network to obtain a fused feature map at each output scale;

[0093] The third determination unit is used to perform target prediction on the fused feature map based on the prediction network and the soft non-maximum suppression Soft-NMS algorithm, and determine the target and target coordinate information included in the video frame to be judged.

[0094] Among them, the multi-scale MobileViTv3 model backbone network includes a MobileViTv3 model, a fast spatial pyramid pooling module, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer. Accordingly, the first determination unit is specifically used for:

[0095] Processing the to-be-judged video frame according to the first convolutional layer, the second convolutional layer, and the MobileViTv3 model to obtain a first feature map at a first output scale;

[0096] Processing the first feature map according to the third convolutional layer and the MobileViTv3 model to obtain a second feature map at a second output scale;

[0097] Processing the second feature map according to the fourth convolutional layer and the MobileViTv3 model to obtain a third feature map at a third output scale;

[0098] The third feature map is processed according to the fifth convolutional layer, the MobileViTv3 model and the fast spatial pyramid pooling module to obtain a fourth feature map at a fourth output scale.

[0099] The neck network includes a first preset target layer, a second preset target layer and an initial target layer. Accordingly, the second determination unit is specifically used for:

[0100] Processing the fourth feature map at the fourth output scale through the initial target layer to obtain a first fused feature map at the fourth output scale;

[0101] Upsampling the fourth feature map through the initial target layer to obtain a fifth feature map and combining it with the third feature map at the third output scale to determine a second fused feature map at the third output scale;

[0102] Upsampling the third feature map through the first preset target layer to obtain a sixth feature map combined with the second feature map at the second output scale to determine a third fused feature map at the second output scale;

[0103] The second feature map is upsampled through the second preset target layer to obtain a seventh feature map, which is combined with the first feature map at the first output scale to determine a fourth fusion map at the first output scale.

[0104] Furthermore, the aggregation behavior determination module 43 is specifically used for:

[0105] Determine the distance between the targets according to the target coordinate information;

[0106] Determine the number of targets that meet the aggregation condition according to each of the distance values ​​and the preset threshold radius;

[0107] If the target quantity is greater than a preset quantity threshold, it is determined that there is a gathering behavior;

[0108] Otherwise, it is determined that there is no aggregation behavior.

[0109] Optionally, the device further comprises:

[0110] The behavior alarm module is used to generate alarm information and notify related objects when aggregation behavior exists after determining whether there is aggregation behavior based on the target coordinate information and the preset quantity threshold.

[0111] The gathering behavior detection device provided in the embodiment of the present invention can execute the gathering behavior detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0112] Embodiment 4

[0113] Figure 5 A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0114] like Figure 5 As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0115] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as the aggregation behavior detection method.

[0117] In some embodiments, the aggregation behavior detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the aggregation behavior detection method described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to perform the aggregation behavior detection method in any other appropriate manner (e.g., by means of firmware).

[0118] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0120] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0123] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0124] In one embodiment, the embodiment of the present invention further includes a computer program product, the computer program product includes a computer program, and the computer program implements the aggregation behavior detection method of any embodiment of the present invention when executed by a processor.

[0125] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0126] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0127] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting aggregation behavior, characterized in that: include: Obtain a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer MobileViTv3 model backbone network, a prediction network, and a neck network including a preset target layer; Performing target detection on the to-be-judged video frame according to the preset target detection model to determine the target and target coordinate information included in the to-be-judged video frame; Determine whether there is aggregation behavior based on the target coordinate information and a preset quantity threshold.

2. The method according to claim 1, characterized in that The performing target detection on the to-be-determined video frame according to the preset target detection model to determine the target and target coordinate information included in the to-be-determined video frame includes: Perform feature extraction on the video frame to be judged according to the multi-scale MobileViTv3 model backbone network, and determine feature maps at different output scales; Performing fusion processing on the feature maps according to the neck network to obtain fused feature maps at each output scale; Based on the prediction network and the soft non-maximum suppression Soft-NMS algorithm, target prediction is performed on the fused feature map to determine the target and target coordinate information included in the video frame to be judged.

3. The method according to claim 2, characterized in that The multi-scale MobileViTv3 model backbone network includes a MobileViTv3 model, a fast spatial pyramid pooling module, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer. Correspondingly, the feature extraction of the video frame to be judged is performed according to the multi-scale MobileViTv3 model backbone network to determine the feature maps at different output scales, including: Processing the to-be-judged video frame according to the first convolutional layer, the second convolutional layer, and the MobileViTv3 model to obtain a first feature map at a first output scale; Processing the first feature map according to the third convolutional layer and the MobileViTv3 model to obtain a second feature map at a second output scale; Processing the second feature map according to the fourth convolutional layer and the MobileViTv3 model to obtain a third feature map at a third output scale; The third feature map is processed according to the fifth convolutional layer, the MobileViTv3 model and the fast spatial pyramid pooling module to obtain a fourth feature map at a fourth output scale.

4. The method according to claim 2, characterized in that: The neck network includes a first preset target layer, a second preset target layer and an initial target layer. Accordingly, the fusion processing is performed on each of the feature maps according to the neck network to obtain a fused feature map at each output scale, including: Processing the fourth feature map at the fourth output scale through the initial target layer to obtain a first fused feature map at the fourth output scale; Upsampling the fourth feature map through the initial target layer to obtain a fifth feature map and combining it with the third feature map at the third output scale to determine a second fused feature map at the third output scale; Upsampling the third feature map through the first preset target layer to obtain a sixth feature map combined with the second feature map at the second output scale to determine a third fused feature map at the second output scale; The second feature map is upsampled through the second preset target layer to obtain a seventh feature map, which is combined with the first feature map at the first output scale to determine a fourth fusion map at the first output scale.

5. The method according to claim 1, characterized in that The determining whether there is a gathering behavior according to the target coordinate information and the preset quantity threshold includes: Determine the distance between the targets according to the target coordinate information; Determine the number of targets that meet the aggregation condition according to each of the distance values ​​and the preset threshold radius; If the target quantity is greater than a preset quantity threshold, it is determined that there is a gathering behavior; Otherwise, it is determined that there is no aggregation behavior.

6. The method according to claim 1, characterized in that After determining whether there is a gathering behavior according to the target coordinate information and the preset quantity threshold, the method further includes: When aggregation behavior occurs, an alarm message is generated and the associated objects are notified.

7. A gathering behavior detection device, characterized in that: include: A video frame acquisition module is used to acquire a video frame to be judged and a preset target detection model, wherein the preset target detection model includes a multi-scale mobile device visual transformer MobileViTv3 model backbone network, a prediction network, and a neck network containing a preset target layer; A target detection module, used to perform target detection on the video frame to be judged according to the preset target detection model, and determine the target and target coordinate information included in the video frame to be judged; The aggregation behavior determination module is used to determine whether there is aggregation behavior based on the target coordinate information and a preset quantity threshold.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the aggregation behavior detection method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the aggregation behavior detection method described in any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the aggregation behavior detection method according to any one of claims 1 to 6.

Images