Tailing behavior monitoring method, device, equipment, medium and product
By combining RGB images and depth images at the top view angle, as well as light curtain detection at the side view angle, and fusing pedestrian detection results, the problem of insufficient monitoring accuracy in traditional trailing behavior monitoring methods is solved, achieving higher monitoring accuracy and visual evidence retention.
Patent Information
- Application Number
- CN202411677092.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-02-28
AI Technical Summary
Traditional trailing behavior monitoring methods have problems such as prone to false alarms, many missed reports and no visual evidence retention, resulting in insufficient monitoring accuracy.
By acquiring the RGB image and depth image of the target area from a preset top view angle, a first pedestrian detection result is generated based on these images, and a light curtain detection is performed from a preset side view angle to obtain the second pedestrian detection result. The two are then fused to determine the location and number of pedestrians in the target area, and trailing behavior monitoring is ultimately performed.
This method improves the accuracy of trailing behavior monitoring, reduces false positives and missed reports, and provides visual evidence through multi-angle detection.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of channel security inspection technology, and in particular to a method, device, equipment, medium and product for monitoring tailgating behavior. Background Art
[0002] With the increasing frequency of international trade and personnel exchanges, the flow of people in customs, border inspection and other places has increased significantly. In order to improve the security of customs, border inspection and other places, it is necessary to monitor the tailgating behavior in the passenger inspection channels of customs, border inspection and other places to prevent people from following and breaking in.
[0003] Traditional tailgating behavior monitoring uses infrared methods, which have problems such as easy false alarms, many missed alarms, and no visual evidence. Therefore, how to improve the accuracy of tailgating behavior monitoring has become an urgent problem to be solved. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a method, device, equipment, medium and product for monitoring tailgating behavior, which can improve the accuracy of tailgating behavior monitoring results to a certain extent.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] According to a first aspect of an embodiment of the present application, a method for monitoring tailgating behavior is provided, the method comprising:
[0007] Acquire the RGB image and depth image of the target area from a preset bird's-eye view angle;
[0008] Based on the RGB image and the depth image, generating a first pedestrian detection result for the target area through a detection model;
[0009] Performing light curtain detection on the target area from a preset side view angle to obtain a second pedestrian detection result for the target area;
[0010] fusing the first pedestrian detection result and the second pedestrian detection result to obtain the positions and number of pedestrians in the target area;
[0011] According to the positions and number of pedestrians in the target area, the target area is monitored for tailgating behavior to obtain a tailgating behavior monitoring result for the target area.
[0012] In some embodiments, the detection model includes a segmentation model and a recognition model; the generating a first pedestrian detection result for the target area by the detection model based on the RGB image and the depth image includes:
[0013] Extracting first image features of the RGB image and second image features of the depth image respectively through a segmentation model, and obtaining fused image features including the first image features and the second image features;
[0014] The first pedestrian detection result is generated by performing nonlinear transformation processing on the fused image features through a recognition model.
[0015] In some embodiments, the segmentation model includes an image encoder, a transformer module, a convolution module, and a mask decoder; the first image feature of the RGB image and the second image feature of the depth image are respectively extracted by the segmentation model to obtain a fused image feature including the first image feature and the second image feature, including:
[0016] By means of the image encoder, the RGB image is converted into a first image block for embedding, and the depth image is converted into a second image block for embedding;
[0017] Performing feature extraction on the first image block embedding by the transformer module to obtain first image features; performing feature extraction on the second image block embedding by the convolution module to obtain second image features;
[0018] The first image feature and the second image feature are fused through the mask decoder to obtain a fused image feature.
[0019] In some embodiments, the recognition model includes a main module and a refinement module; the step of performing nonlinear transformation processing on the fused image features through the recognition model to generate the first pedestrian detection result includes:
[0020] Performing nonlinear transformation processing on the fused image features through the main module to generate an initial pedestrian detection result corresponding to the fused image features;
[0021] The refinement module performs nonlinear transformation processing on the fused image features and the initial pedestrian detection result to generate a first pedestrian detection result corresponding to the fused image features.
[0022] In some embodiments, performing nonlinear transformation processing on the fused image features by the main module to generate an initial pedestrian detection result corresponding to the fused image features includes:
[0023] Inputting the fused image features into the main module, extracting the fused image features through a convolutional neural network in the main module, and obtaining main fused image features;
[0024] Performing convergence processing on the main fused image features through a batch normalization layer in the main module to obtain a main converged feature;
[0025] The main convergence feature is subjected to nonlinear transformation processing through the activation function in the main module to generate the initial pedestrian detection result.
[0026] In some embodiments, performing nonlinear transformation processing on the fused image features and the initial pedestrian detection result by the refinement module to generate a first pedestrian detection result corresponding to the fused image features includes:
[0027] Inputting the fused image features and the initial pedestrian detection results into the refinement module, and extracting features from the fused image features and the initial pedestrian detection results through a convolutional neural network in the refinement module to obtain refined fused image features;
[0028] Performing convergence processing on the refined fusion image features through a batch normalization layer in the refinement module to obtain refined converged features;
[0029] The refined convergence features are processed by nonlinear transformation through the activation function in the refinement module to generate the first pedestrian detection result.
[0030] In some embodiments, performing tailgating behavior monitoring on the target area according to the positions and number of pedestrians in the target area to obtain the tailgating behavior monitoring result of the target area includes:
[0031] When the number of pedestrians within a preset range of at least one pedestrian position in the target area is greater than 1, the tailgating behavior monitoring result indicates that tailgating behavior exists in the target area;
[0032] When the number of pedestrians within the preset range of each pedestrian position in the target area is 1, the tailgating behavior monitoring result indicates that there is no tailgating behavior in the target area.
[0033] According to a second aspect of an embodiment of the present application, a tailgating behavior monitoring device is provided, comprising:
[0034] An acquisition module is used to acquire an RGB image and a depth image of a target area from a preset bird's-eye view angle;
[0035] A first detection module, configured to generate a first pedestrian detection result for the target area through a detection model based on the RGB image and the depth image;
[0036] A second detection module, used to perform light curtain detection on the target area from a preset side view angle to obtain a second pedestrian detection result for the target area;
[0037] A fusion module, used for fusing the first pedestrian detection result and the second pedestrian detection result to obtain the position and number of pedestrians in the target area;
[0038] The monitoring module is used to perform tailgating behavior monitoring on the target area according to the positions and number of pedestrians in the target area, and obtain the tailgating behavior monitoring result of the target area.
[0039] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:
[0040] A memory for storing executable instructions;
[0041] The processor is used to implement the tailgating behavior monitoring method as described in the first aspect above when executing the executable instructions stored in the memory.
[0042] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, storing executable instructions for implementing the tailgating behavior monitoring method as described in the first aspect above when executed by a processor.
[0043] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implements the tailgating behavior monitoring method as described in the first aspect above.
[0044] In an embodiment of the present application, an RGB image and a depth image of a target area are acquired from a preset top-down angle; a first pedestrian detection result for the target area is generated through a detection model based on the RGB image and the depth image; a light curtain detection is performed on the target area from a preset side-view angle to obtain a second pedestrian detection result for the target area; the first pedestrian detection result and the second pedestrian detection result are fused to obtain the positions and the number of pedestrians in the target area; and according to the positions and the number of pedestrians in the target area, a following behavior monitoring is performed on the target area to obtain a following behavior monitoring result for the target area.
[0045] In the embodiment of the present application, the following behavior monitoring result is determined by combining the first pedestrian detection result generated based on the RGB image and the depth image from the top-down angle and the second pedestrian detection result generated by the light curtain detection from the side-view angle. Combining images from multiple angles can improve the accuracy of distinguishing between people and objects, thereby reducing false alarms and missed alarms, and improving the accuracy of the following behavior monitoring results.
[0046] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0048] Figure 1 Schematic diagram of the implementation process of the tailgating behavior monitoring method applicable to the embodiment of the present application Figure 1 ;
[0049] Figure 2 Schematic diagram of the implementation process of the tailgating behavior monitoring method applicable to the embodiment of the present application Figure 2 ;
[0050] Figure 3 Schematic diagram of the implementation process of the tailgating behavior monitoring method applicable to the embodiment of the present application Figure 3 ;
[0051] Figure 4 A schematic diagram of the structure of a tailgating behavior monitoring device applicable to an embodiment of the present application;
[0052] Figure 5 A schematic diagram of a hardware entity of an electronic device applicable to an embodiment of the present application. DETAILED DESCRIPTION
[0053] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.
[0054] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in the embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.
[0055] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0056] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.
[0057] In addition, the term "and / or" in the embodiments of the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0058] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.
[0059] In order to improve the security of customs and border inspection places, it is necessary to monitor the tailgating behavior in the passenger inspection channels of customs and border inspection places to prevent people from following and breaking in. The tailgating behavior monitoring methods in related technologies have problems such as easy false alarms, many missed alarms, and no visual evidence retention.
[0060] An embodiment of the present application provides a tailgating behavior monitoring method, which determines a tailgating behavior monitoring result by combining a first pedestrian detection result generated based on an RGB image and a depth image from a top-down angle and a second pedestrian detection result generated by light curtain detection from a side-view angle. Combining images from multiple angles can improve the accuracy of distinguishing between people and objects, thereby reducing false alarms and missed alarms, and improving the accuracy of the tailgating behavior monitoring results.
[0061] The specific implementation of the embodiment of the present application is described below in conjunction with the accompanying drawings of the embodiment of the present application.
[0062] See also Figure 1 , Figure 1 Schematic diagram of the implementation process of the tailgating behavior monitoring method applicable to the embodiment of the present application Figure 1 The following behavior monitoring method of the embodiment of the present application comprises the following steps:
[0063] Step S101: Acquire an RGB image and a depth image of a target area from a preset bird's-eye view angle.
[0064] Specifically, the preset top-down angle in the embodiment of the present application refers to the perspective of looking down from directly above the target area. The target area may be the area corresponding to the inspection channel of customs, border inspection and other places. The specific range or size of the target area can be set according to actual conditions. The present application does not impose any special restrictions on the range or size of the target area. The RGB image is a color model that uses three color channels, red (Red), green (Green), and blue (Blue), to represent an image. A depth image refers to an image that uses the depth value of each point in the target area as a pixel value, which can reflect the geometric shape of objects in the target area.
[0065] Exemplarily, an image acquisition device for acquiring RGB images and an image acquisition device for acquiring depth images may be set directly above the target area. When a pedestrian enters the target area, the RGB image and depth image of the target area are acquired by the image acquisition device for acquiring RGB images and the image acquisition device for acquiring depth images.
[0066] Optionally, the acquisition frequency of the image acquisition device may be set according to actual needs, and the RGB image and the depth image of the target area may be acquired according to the acquisition frequency.
[0067] Step S102: Based on the RGB image and the depth image, a first pedestrian detection result for the target area is generated by a detection model.
[0068] Specifically, the detection model may be a model composed of a convolutional neural network and other parts. The first pedestrian detection result may include detection results of pedestrian positions and the number of pedestrians, and the first pedestrian detection result may be presented in the form of a binary confidence map or in other forms. Taking the form of a binary confidence map as an example, the number of head center points and the position of the head center point in the binary confidence map may be used to determine the position and number of pedestrians in the binary confidence map.
[0069] Exemplarily, after the RGB image and the depth image are acquired, the RGB image and the depth image may be input into a detection model to generate a first pedestrian detection result through the detection model.
[0070] Step S103: Perform light curtain detection on the target area from a preset side viewing angle to obtain a second pedestrian detection result for the target area.
[0071] Specifically, the preset side view angle in the embodiment of the present application refers to the viewing angle from one side of the target area. Taking the target area as a certain inspection channel of the border inspection as an example, the preset side view angle can be the angle from the gate on one side of the inspection channel to the gate on the other side. The second pedestrian detection result can include the detection results of the pedestrian position and the number of pedestrians, and the second pedestrian detection result can be presented in the form of images or data.
[0072] Exemplarily, a light curtain detection may be performed on the target area from one side of the target area to obtain a light curtain detection result for the target area, that is, a second pedestrian detection result.
[0073] Step S104: Fuse the first pedestrian detection result and the second pedestrian detection result to obtain the positions and number of pedestrians in the target area.
[0074] Specifically, the position of the pedestrian can be represented by a coordinate point.
[0075] Exemplarily, the shapes of pedestrians or objects appearing in the target area can be more intuitively seen through the second pedestrian detection result. Fusion of the first pedestrian detection result and the second pedestrian detection result can refer to: adjusting the positions and number of pedestrians in the first pedestrian detection result by non-pedestrian factors such as luggage and pets in the second pedestrian detection result to obtain a more accurate first pedestrian detection result, thereby determining the positions and number of pedestrians in the target area based on the more accurate first pedestrian detection result. Fusion of the first pedestrian detection result and the second pedestrian detection result can also refer to: screening out the coordinate points where the positions of pedestrians in the first pedestrian detection result and the second pedestrian detection result overlap, determining the positions of pedestrians in the target area based on the coordinate points where the positions of the pedestrians overlap, thereby determining the positions and number of pedestrians in the target area. The embodiment of the present application does not specifically limit the method of fusing the first pedestrian detection result and the second pedestrian detection result.
[0076] Step S105: perform tailgating behavior monitoring on the target area according to the positions and number of pedestrians in the target area, and obtain a tailgating behavior monitoring result of the target area.
[0077] Specifically, the tailing behavior monitoring result is used to characterize whether there is tailing behavior in the target area.
[0078] For example, each pedestrian position can be detected to determine whether there are other pedestrians within a certain range of each pedestrian position. When there are other pedestrians within a certain range of a pedestrian position, the tailing behavior monitoring result indicates that there is tailing behavior in the target area; when there are no other pedestrians within a certain range of each pedestrian position, the tailing behavior monitoring result indicates that there is no tailing behavior in the target area. The certain range of the pedestrian position can be the range corresponding to the circular area with the pedestrian position as the center and the minimum distance as the radius, wherein the minimum distance can be 5 cm, 10 cm or other lengths set according to actual needs.
[0079] In the embodiment of the present application, the following behavior monitoring result is determined by combining the first pedestrian detection result generated based on the RGB image and the depth image from the top-down angle and the second pedestrian detection result generated by the light curtain detection from the side-view angle. Combining images from multiple angles can improve the accuracy of distinguishing between people and objects, thereby reducing false alarms and missed alarms, and improving the accuracy of the following behavior monitoring results.
[0080] Optionally, in some embodiments, the detection model includes a segmentation model and a recognition model; the segmentation model can be used to extract first image features of the RGB image and second image features of the depth image, respectively, to obtain fused image features including the first image features and the second image features; the recognition model can be used to perform nonlinear transformation processing on the fused image features to generate a first pedestrian detection result.
[0081] Optionally, in some embodiments, the segmentation model includes an image encoder, a transformer module, a convolution module and a mask decoder; the first image features of the RGB image and the second image features of the depth image are respectively extracted through the segmentation model to obtain a fused image feature containing the first image feature and the second image feature, including: through the image encoder, the RGB image is converted into a first image block embedding, and the depth image is converted into a second image block embedding; through the transformer module, feature extraction is performed on the first image block embedding to obtain the first image feature; through the convolution module, feature extraction is performed on the second image block embedding to obtain the second image feature; through the mask decoder, the first image feature and the second image feature are fused to obtain the fused image feature.
[0082] Specifically, the converter module in the embodiment of the present application includes a multi-head self-attention layer and a feed-forward neural network. The convolution module in the embodiment of the present application can be a lightweight convolution module.
[0083] For example, see Figure 2 , the RGB image and the depth image can be input into the segmentation model, and the RGB image can be converted into the first image block embedding through the image encoder in the segmentation model, and the depth image can be converted into the second image block embedding. The transformer module in the segmentation model extracts the features of the first image block embedding to obtain the first image features; the convolution module in the segmentation model extracts the features of the second image block embedding to obtain the second image features, and the mask decoder fuses the first image features and the second image features to obtain the fused image features. Figure 2 As shown, the second image feature extracted by the convolution module can also be connected in parallel to the first image block embedding, and then the first image block embedding fused with the second image feature is subjected to feature extraction through multiple transformer modules and convolution modules to obtain a fused image feature fused with the first image feature and the second image feature, and then the fused image feature is output through a mask decoder.
[0084] Alternatively, if Figure 2 As shown, in some of the embodiments, while the RGB image is input into the segmentation model, the target detection algorithm can be used to perform target detection on the RGB image to determine the edge box of the object in the RGB image, and the edge box of the object is input into the segmentation model as a prompt word. The segmentation model encodes the prompt word to generate embedded information of the edge box prompt word, and the embedded information and the fused image features are input into the mask decoder to fuse the embedded information into the fused image features through the mask decoder, thereby improving the accuracy of the fused image features.
[0085] In the embodiment of the present application, a fused image feature that fuses the first image feature and the second image feature of the RGB image and the depth image is generated by a segmentation model, and the accuracy of identifying objects in the image can be improved according to the fused image feature.
[0086] Optionally, in some embodiments, the recognition model includes a main module and a refinement module; the recognition model performs nonlinear transformation processing on the fused image features to generate a first pedestrian detection result, including: performing nonlinear transformation processing on the fused image features through the main module to generate an initial pedestrian detection result corresponding to the fused image features; performing nonlinear transformation processing on the fused image features and the initial pedestrian detection result through the refinement module to generate a first pedestrian detection result corresponding to the fused image features.
[0087] Specifically, the main module and the refinement module of the input recognition model both include an encoder module and a decoder module. The first pedestrian detection result can be presented in the form of a binary confidence map or in other forms.
[0088] For example, see Figure 2 The fused image features can be input into the main module of the recognition model, and the fused image features can be processed by the encoder module and the decoder module in the main module for nonlinear transformation, so as to generate the initial pedestrian detection result corresponding to the fused image features. The fused image features and the initial pedestrian detection result can be input into the refinement module of the recognition model, and the fused image features and the initial pedestrian detection result can be processed by the encoder module and the decoder module in the refinement module for nonlinear transformation, so as to generate the first pedestrian detection result corresponding to the fused image features.
[0089] Alternatively, if Figure 2As shown, in some embodiments, after obtaining the second pedestrian detection result and the initial pedestrian detection result, the second pedestrian detection result, the fused image features and the initial pedestrian detection result can be input into the refinement module of the recognition model, so as to perform nonlinear transformation processing on the second pedestrian detection result, the fused image features and the initial pedestrian detection result through the encoder module and the decoder module in the refinement module to generate a first pedestrian detection result, so as to determine the pedestrian position and the number of pedestrians in the target area according to the first pedestrian detection result integrated with the second pedestrian detection result, and at the same time improve the accuracy of the determined pedestrian position and the number of pedestrians in the target area.
[0090] Optionally, in some embodiments, the process of recognizing model training is as follows: the main module uses function F m =(p i ,θ m ) indicates that p i represents the input fusion image feature, p i Normalized in the interval (0,1), the vector θ m Represents the trainable parameters of the main module; the refinement module uses function F r =(p i ,q i ,θ r ) indicates that q i represents the confidence map corresponding to the first pedestrian detection result, q i Normalized in the interval (0,1). Confidence map q i From the covariance matrix M = σΠ 2*2 The two-dimensional Gaussian distribution and generation are centered on the coordinates corresponding to the positions of each pedestrian in the target area. σ can be 6 pixels, and the vector θ r Represents the trainable parameters of the refinement module. The trainable parameters of the recognition model are expressed as θ = (θ m ,θ r ).
[0091] The function of the recognition model is expressed as:
[0092] F(p i ,θ)=F r (p i , F m (p i ,θ m ), θ r )
[0093] A set of corresponding fused image features and the confidence map corresponding to the first pedestrian detection result {(p i ,q i )}i=1,...,N to train the recognition model to minimize the loss function. The loss function is determined as follows:
[0094]
[0095] in, represents the hyperparameter used for weighting, F(p i ,θ)-q i is the difference between the function of the recognition model and the confidence map corresponding to the first pedestrian detection result, F m (p i ,θ n )-q r It is the difference between the confidence map corresponding to the refinement module function and the first pedestrian detection result.
[0096] In an embodiment of the present application, a first pedestrian detection result is generated by a recognition model so that the positions and number of pedestrians can be viewed more intuitively based on the first pedestrian detection result.
[0097] Optionally, in some of the embodiments, the fused image features are subjected to nonlinear transformation processing by the main module to generate initial pedestrian detection results corresponding to the fused image features, including: inputting the fused image features into the main module, extracting features of the fused image features by the convolutional neural network in the main module to obtain main fused image features; performing convergence processing on the main fused image features by the batch normalization layer in the main module to obtain main converged features; performing nonlinear transformation processing on the main converged features by the activation function in the main module to generate initial pedestrian detection results.
[0098] Specifically, the main module of the recognition model in the embodiment of the present application includes a convolutional neural network, a batch normalization layer, an activation function, etc. Among them, the convolutional neural network may include multiple convolutional layers.
[0099] Optionally, in some of the embodiments, the main module of the recognition model includes an encoder and a decoder, the encoder includes three convolutional layers, the decoder includes three separable convolutional layers, and the main module also includes an activation function ReLU, a batch normalization layer, a cropping layer, and an upsampling layer. Accordingly, the fused image features are processed by the main module through a nonlinear transformation to generate an initial pedestrian detection result corresponding to the fused image features, including: taking the fused image feature size of the input main module as 256×256×1 pixels, the convolution layer output feature size as 128×128×64 pixels, the convolution kernel as 7×7, the step size as 2, the maximum pooling layer output as 42×42×64 pixels, and the filter as 3×3 as an example, the fused image features are first input into the encoder, and the first convolution of the encoder is The output size of the convolution layer is 42×42×256 pixels, the convolution kernel is 3×3, the stride is 1, a=64, b=64, c=256, the output size of the second convolution layer of the encoder is 21×21×512 pixels, the convolution kernel is 3×3, the stride is 2, a=128, b=128, c=512, the output size of the third convolution layer of the encoder is 11×11×1024 pixels, the convolution kernel is 3×3, the stride is 2, a=256, b=256, c=1024. Next, the fused image features processed by the encoder can be input into the decoder. The output size of the first separable convolutional layer of the decoder is 11×11×256 pixels, the convolution kernel is 3×3, the step size is 1, a=1024, b=1024, c=256, the output size of the second separable convolutional layer of the decoder is 22×22×128 pixels, the convolution kernel is 3×3, the step size is 2, a=512, b=512, c=128, and the output size of the third separable convolutional layer of the decoder is 44×44×64 pixels, the convolution kernel is 3×3, the step size is 2, a=256, b=256, c=64. Next, the size of the fused image features processed by the decoder is cropped through the first cropping layer. The output of the first cropping layer is 43×43×64 pixels, and the acquisition parameters are [(1, 0), (1, 0)]. The first set of parameters represents the cropped pixels on the left, 44-2=42, and the second set of parameters represents the cropping of 1 pixel on the upper side. Next, the size of the cropped fused image features can be enlarged through the upsampling layer, and the output is 129×129×64 pixels. The sampling parameters of the upsampling layer are 3×3, which is expanded by 3 times. Next, the sampled fused image features are input into the convolution layer, and the convolution layer outputs the fused image features with a size of 258×258×64 pixels, a convolution kernel of 7×7, and a step size of 2. Next, the fused image features output by the convolution layer are input into the second cropping layer, and the second cropping layer outputs fused image features of 256×256×64 pixels. The acquisition parameters of the second cropping layer are [(1, 1), (1, 1)].Next, the fused image features output by the second crop layer are processed in turn through a batch normalization layer, an activation function ReLU, a convolution layer with an output size of 256×256×1 pixels, a convolution kernel of 3×3, a step size of 1, and an activation function Sigmoid, and a confidence map with a size of 256×256×1 pixels is output, which is the initial pedestrian detection result. It should be noted that in the encoder and decoder, a, b, and c represent the number of filters in the convolution layer in the branch.
[0100] In the embodiment of the present application, the robustness of the initial pedestrian detection result can be improved by processing the fused image features through multiple convolutional layers in the main module.
[0101] Optionally, in some of the embodiments, a refinement module is used to perform nonlinear transformation processing on the fused image features and the initial pedestrian detection results to generate a first pedestrian detection result corresponding to the fused image features, including: inputting the fused image features and the initial pedestrian detection results into the refinement module, performing feature extraction on the fused image features and the initial pedestrian detection results through a convolutional neural network in the refinement module to obtain refined fused image features; performing convergence processing on the refined fused image features through a batch normalization layer in the refinement module to obtain refined converged features; performing nonlinear transformation processing on the refined converged features through an activation function in the refinement module to generate the first pedestrian detection result.
[0102] Specifically, the refinement module of the recognition model in the embodiment of the present application includes a convolutional neural network, a batch normalization layer, an activation function, etc. Among them, the convolutional neural network may include multiple convolutional layers.
[0103] Optionally, in some of the embodiments, the refinement module of the recognition model includes an encoder and a decoder, the encoder includes two convolutional layers, the decoder includes two separable convolutional layers, and the refinement module also includes an activation function ReLU, a maximum pooling layer, a zero padding layer, a cropping layer, and an upsampling layer. Accordingly, the fused image features and the initial pedestrian detection results are processed by the refinement module through nonlinear transformation to generate a first pedestrian detection result corresponding to the fused image features, including: taking the fused image feature size of the input refinement module as 256×256×1 pixels, the convolution layer output feature size as 128×128×64 pixels, the convolution kernel as 7×7, the step size as 2, the maximum pooling layer output as 42×42×64 pixels, and the filter as 3×3 as an example, the first convolution layer output size of the encoder is 42×42×256 pixels, the convolution kernel is 3×3, the step size is 1, a=64, b=64, c=256, and the second convolution layer output size ... The output size of the convolution layer is 21×21×512 pixels, the convolution kernel is 3×3, the stride is 2, a=128, b=128, c=512; next, the output size of the first separable convolution layer of the decoder is 42×42×128 pixels, the convolution kernel is 3×3, the stride is 2, a=512, b=512, c=128, the output size of the second separable convolution layer is 84×84×64 pixels, the convolution kernel is 3×3, the stride is 2, a=256, b=256, c=64; the output of the zero padding layer is 86×86×64 pixels, the padding parameter is (1, 1), and the feature is padded with 1 bit of 0 on the top, bottom, left and right sides. The output of the upsampling layer is 258×258×64 pixels, the sampling parameter is 3×3, and the expansion is 3 times. The cropping layer output is 256×256×64 pixels, and the acquisition parameters are [(1, 1), (1, 1)]. The first set of parameters represents the left and right cropping of 1, 258-2=256, and the second set of parameters represents the top and bottom cropping of 1. The convolution layer output size is 256×256×64 pixels, the convolution kernel is 3×3, and the step size is 1. The activation function Sigmoid output size is 256×256×1 pixel. The above parts of the refinement module are used to process the input first pedestrian detection result and the fused image features respectively to output a confidence map of size 256×256×1 pixel, i.e., the first pedestrian detection result.
[0104] In the embodiment of the present application, the first pedestrian detection result and the fused image features are processed by the refinement module to obtain the first pedestrian detection result with higher accuracy and precision.
[0105] Optionally, in some of the embodiments, the target area is monitored for tailgating behavior based on the positions and number of pedestrians in the target area to obtain a tailgating behavior monitoring result for the target area, including: when the number of pedestrians within a preset range of at least one pedestrian position in the target area is greater than 1, the tailgating behavior monitoring result indicates that there is tailing behavior in the target area; when the number of pedestrians within the preset range of each pedestrian position in the target area is 1, the tailing behavior monitoring result indicates that there is no tailing behavior in the target area.
[0106] Specifically, the preset range of the pedestrian's position refers to a circular range with the pedestrian's position as the center and a preset distance as the radius, wherein the preset distance may be 5 cm, 10 cm, or other lengths set according to actual needs. The tailing behavior monitoring result is used to characterize whether there is tailing behavior in the target area.
[0107] Exemplarily, when the number of pedestrians within a preset range of a certain pedestrian position in the target area is greater than 1, there is a following behavior at the pedestrian position in the target area, and the monitoring result indicates that there is a following behavior in the target area; when the number of pedestrians within the preset range of each pedestrian position in the target area is 1, there is no following behavior at each pedestrian position in the target area, and the following behavior monitoring result indicates that there is no following behavior in the target area.
[0108] See also Figure 3 , Figure 3 Schematic diagram of the implementation process of the tailgating behavior monitoring method applicable to the embodiment of the present application Figure 3 .
[0109] The following steps are included in the following steps:
[0110] 1. Pedestrians enter the inspection channel.
[0111] The inspection channel corresponds to the target area in the above embodiment.
[0112] In an embodiment of the present application, the status of a pedestrian entering an inspection channel can be identified.
[0113] 2. Use light curtain for horizontal detection.
[0114] The horizontal direction corresponds to the preset side viewing angle in the above embodiment.
[0115] When a pedestrian is identified entering the inspection channel, the lateral angle of the inspection channel can be detected by the light curtain.
[0116] 3. Use RGB camera for vertical detection.
[0117] The vertical direction corresponds to the preset top-down angle in the above embodiment.
[0118] The longitudinal angle of the inspection channel can be detected by an RGB camera.
[0119] 4. Use depth camera for vertical detection.
[0120] The longitudinal angle of the inspection channel can be detected by a depth camera.
[0121] 5. Comprehensively judge and check whether there is any tailing behavior in the channel.
[0122] Combining the detection results obtained by the light curtain horizontal detection, RGB camera vertical detection and depth camera vertical detection, a comprehensive judgment is made on whether there is any tailing behavior in the inspection channel.
[0123] In the embodiment of the present application, the detection results obtained by combining the horizontal detection of the light curtain, the vertical detection of the RGB camera and the vertical detection of the depth camera are used to determine whether there is a tailing behavior in the inspection channel. Combining multiple detection results can improve the accuracy of distinguishing people and objects, thereby reducing false positives and missed positives, and improving the accuracy of tailing behavior judgment.
[0124] See also Figure 4 , Figure 4 The following is a schematic diagram of the structure of a tailgating behavior monitoring device applicable to an embodiment of the present application. The tailgating behavior monitoring device of this embodiment includes the following modules:
[0125] An acquisition module 401 is used to acquire an RGB image and a depth image of a target area from a preset bird's-eye view angle;
[0126] A first detection module 402, configured to generate a first pedestrian detection result for a target area through a detection model based on the RGB image and the depth image;
[0127] The second detection module 403 is used to perform light curtain detection on the target area from a preset side view angle to obtain a second pedestrian detection result for the target area;
[0128] A fusion module 404 is used to fuse the first pedestrian detection result and the second pedestrian detection result to obtain the position and number of pedestrians in the target area;
[0129] The monitoring module 405 is used to perform tailgating behavior monitoring on the target area according to the positions and number of pedestrians in the target area, and obtain the tailgating behavior monitoring result of the target area.
[0130] Optionally, in some embodiments, the detection model includes a segmentation model and a recognition model; the first detection module 402 is specifically used to: extract the first image features of the RGB image and the second image features of the depth image respectively through the segmentation model to obtain a fused image feature including the first image feature and the second image feature; perform nonlinear transformation processing on the fused image feature through the recognition model to generate a first pedestrian detection result.
[0131] Optionally, in some embodiments, the segmentation model includes an image encoder, a transformer module, a convolution module and a mask decoder; the first detection module 402 is specifically used to: convert the RGB image into a first image block embedding and convert the depth image into a second image block embedding through the image encoder; perform feature extraction on the first image block embedding through the transformer module to obtain a first image feature; perform feature extraction on the second image block embedding through the convolution module to obtain a second image feature; and fuse the first image feature and the second image feature through the mask decoder to obtain a fused image feature.
[0132] Optionally, in some embodiments, the recognition model includes a main module and a refinement module; the first detection module 402 is specifically used to: perform nonlinear transformation processing on the fused image features through the main module to generate an initial pedestrian detection result corresponding to the fused image features; perform nonlinear transformation processing on the fused image features and the initial pedestrian detection result through the refinement module to generate a first pedestrian detection result corresponding to the fused image features.
[0133] Optionally, in some of the embodiments, the first detection module 402 is specifically used to: input the fused image features into the main module, perform feature extraction on the fused image features through the convolutional neural network in the main module to obtain main fused image features; perform convergence processing on the main fused image features through the batch normalization layer in the main module to obtain main converged features; perform nonlinear transformation processing on the main converged features through the activation function in the main module to generate initial pedestrian detection results.
[0134] Optionally, in some of the embodiments, the first detection module 402 is specifically used to: input the fused image features and the initial pedestrian detection results into a refinement module, perform feature extraction on the fused image features and the initial pedestrian detection results through a convolutional neural network in the refinement module to obtain refined fused image features; perform convergence processing on the refined fused image features through a batch normalization layer in the refinement module to obtain refined converged features; perform nonlinear transformation processing on the refined converged features through an activation function in the refinement module to generate a first pedestrian detection result.
[0135] Optionally, in some of the embodiments, the monitoring module 405 is specifically used to: when the number of pedestrians within a preset range of at least one pedestrian position in the target area is greater than 1, the following behavior monitoring result indicates that there is a following behavior in the target area; when the number of pedestrians within the preset range of each pedestrian position in the target area is 1, the following behavior monitoring result indicates that there is no following behavior in the target area.
[0136] The tailing behavior monitoring device of this embodiment is used to implement the corresponding tailing behavior monitoring method in the aforementioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. In addition, the functional implementation of each module in the tailing behavior monitoring device of this embodiment can refer to the description of the corresponding part in the aforementioned method embodiment, which will not be repeated here.
[0137] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.
[0138] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.
[0139] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above computer program and computer program product embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the computer program and computer program product embodiments of this application, please refer to the description of the method embodiments of this application for understanding.
[0140] Figure 5 A schematic diagram of a hardware entity of an electronic device applicable to an embodiment of the present application is shown in FIG. Figure 5 As shown, the hardware entity of the electronic device 500 includes: a processor 501 and a memory 502, wherein the memory 502 stores a computer program that can be run on the processor 501, and the processor 501 implements the steps in the method of any of the above embodiments when executing the program.
[0141] The memory 502 stores computer programs that can be run on the processor. The memory 502 is configured to store instructions and applications executable by the processor 501. It can also cache data to be processed or processed by the processor 501 and various modules in the electronic device 500 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).
[0142] When the processor 501 executes the program, the steps of any of the above-mentioned vehicle abnormal behavior warning methods are implemented. The processor 501 generally controls the overall operation of the electronic device 500.
[0143] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the vehicle abnormal behavior warning method in any of the above embodiments.
[0144] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.
[0145] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.
[0146] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.
[0147] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.
[0148] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0149] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.
[0150] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0151] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0152] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.
[0153] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0154] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0155] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0156] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0157] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for monitoring tailgating behavior, characterized in that: include: Acquire the RGB image and depth image of the target area from a preset bird's-eye view angle; Based on the RGB image and the depth image, generating a first pedestrian detection result for the target area through a detection model; Performing light curtain detection on the target area from a preset side view angle to obtain a second pedestrian detection result for the target area; fusing the first pedestrian detection result and the second pedestrian detection result to obtain the positions and number of pedestrians in the target area; According to the positions and number of pedestrians in the target area, the target area is monitored for tailgating behavior to obtain a tailgating behavior monitoring result for the target area.
2. The method according to claim 1, characterized in that The detection model includes a segmentation model and a recognition model; the generating a first pedestrian detection result for the target area through the detection model based on the RGB image and the depth image includes: Extracting first image features of the RGB image and second image features of the depth image respectively through a segmentation model, and obtaining fused image features including the first image features and the second image features; The first pedestrian detection result is generated by performing nonlinear transformation processing on the fused image features through a recognition model.
3. The method according to claim 2, characterized in that The segmentation model includes an image encoder, a transformer module, a convolution module and a mask decoder; the first image feature of the RGB image and the second image feature of the depth image are respectively extracted by the segmentation model to obtain a fused image feature including the first image feature and the second image feature, including: By means of the image encoder, the RGB image is converted into a first image block for embedding, and the depth image is converted into a second image block for embedding; Performing feature extraction on the first image block embedding by the transformer module to obtain first image features; performing feature extraction on the second image block embedding by the convolution module to obtain second image features; The first image feature and the second image feature are fused through the mask decoder to obtain a fused image feature.
4. The method according to claim 2, characterized in that: The recognition model includes a main module and a refinement module; The step of performing nonlinear transformation processing on the fused image features through a recognition model to generate the first pedestrian detection result includes: Performing nonlinear transformation processing on the fused image features through the main module to generate an initial pedestrian detection result corresponding to the fused image features; The refinement module performs nonlinear transformation processing on the fused image features and the initial pedestrian detection result to generate a first pedestrian detection result corresponding to the fused image features.
5. The method according to claim 4, characterized in that The step of performing nonlinear transformation processing on the fused image features by the main module to generate an initial pedestrian detection result corresponding to the fused image features includes: Inputting the fused image features into the main module, extracting the fused image features through a convolutional neural network in the main module, and obtaining main fused image features; Performing convergence processing on the main fused image features through a batch normalization layer in the main module to obtain a main converged feature; The main convergence feature is subjected to nonlinear transformation processing through the activation function in the main module to generate the initial pedestrian detection result.
6. The method according to claim 4, characterized in that The step of performing nonlinear transformation processing on the fused image features and the initial pedestrian detection result by the refinement module to generate a first pedestrian detection result corresponding to the fused image features includes: Inputting the fused image features and the initial pedestrian detection results into the refinement module, and extracting features from the fused image features and the initial pedestrian detection results through a convolutional neural network in the refinement module to obtain refined fused image features; Performing convergence processing on the refined fusion image features through a batch normalization layer in the refinement module to obtain refined converged features; The refined convergence features are processed by nonlinear transformation through the activation function in the refinement module to generate the first pedestrian detection result.
7. The method according to claim 1, characterized in that The following behavior monitoring is performed on the target area according to the positions and number of pedestrians in the target area to obtain the following behavior monitoring result of the target area, including: When the number of pedestrians within a preset range of at least one pedestrian position in the target area is greater than 1, the tailgating behavior monitoring result indicates that tailgating behavior exists in the target area; When the number of pedestrians within the preset range of each pedestrian position in the target area is 1, the tailgating behavior monitoring result indicates that there is no tailgating behavior in the target area.
8. A tailing behavior monitoring device, characterized in that: include: An acquisition module is used to acquire an RGB image and a depth image of a target area from a preset bird's-eye view angle; A first detection module, configured to generate a first pedestrian detection result for the target area through a detection model based on the RGB image and the depth image; A second detection module, used to perform light curtain detection on the target area from a preset side view angle to obtain a second pedestrian detection result for the target area; A fusion module, used for fusing the first pedestrian detection result and the second pedestrian detection result to obtain the position and number of pedestrians in the target area; The monitoring module is used to perform tailgating behavior monitoring on the target area according to the positions and number of pedestrians in the target area, and obtain the tailgating behavior monitoring result of the target area.
9. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the tailgating behavior monitoring method described in any one of claims 1 to 7 when executing the executable instructions stored in the memory.
10. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the tailgating behavior monitoring method described in any one of claims 1 to 7 when executed by a processor.
11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method for monitoring tailgating behavior described in any one of claims 1 to 7 is implemented.