A multi-scale prediction behavior recognition system and method

Abstract

The invention discloses a multi-scale prediction behavior recognition system, which relates to the field of video understanding. The multi-scale prediction behavior recognition system includes a video decoding module, a video stream preprocessing module, a spatial feature extraction network and a spatiotemporal feature extraction network. The video stream preprocessing module processes the frame image processed by the video decoding module into two parts, which are respectively input into the spatial feature extraction network and the spatiotemporal feature extraction network. The specific steps include: processing the video stream data of the network camera or the video stream data of the local video into the form of a frame image through the video decoding module; the video stream preprocessing module processes the frame image processed by the video decoding module into two parts, which are respectively input into the spatial feature extraction network and the spatiotemporal feature extraction network; the data samples are respectively input into the spatial feature extraction network and the spatiotemporal feature extraction network; and the output feature map is decoded.

Description

Technical Field

[0001] The present invention relates to the field of video understanding, and in particular to a multi-scale prediction behavior recognition system. Background Art

[0002] Video understanding has always been a hot research topic in the field of computer vision. Through video understanding algorithms, video streams are obtained to analyze specific human actions, identify action types, and locate the spatial location of the action. Video understanding action recognition involves a huge amount of computational work in the model, especially the behavior algorithms based on deep learning have increasingly higher requirements for software and hardware. In today's highly intelligent and integrated society, certain requirements are placed on the real-time performance and accuracy of behavior recognition algorithms. Feature fusion at different scales can improve accuracy, but this method has limited improvement in accuracy for different action categories and video content, and cannot adapt to different scenarios. In addition, the single-scale output of the model also limits the accuracy improvement of the model. Therefore, solving the huge challenges faced by the feature fusion method and the multi-scale prediction output method, and promoting the implementation of artificial intelligence products with behavior recognition algorithms will promote social progress.

[0003] The Chinese patent "Video-based behavior recognition method, behavior recognition device and terminal equipment" with patent number 2018107077110 proposes a video-based behavior recognition method, behavior recognition device, terminal equipment and computer-readable storage medium. The method extracts features from the RGB frames of the video through a feature extractor, inputs the proposed features into the long short-term memory network LSTM for time series modeling, and inputs the features of the last convolutional layer into the convolutional LSTM for time series modeling; the features output after modeling are input into the joint optimization layer, and the RGB frames of the video are used for behavior recognition through the joint optimization layer. However, this method has poor recognition accuracy and does not have good generalization ability. It has basically no recognition effect on scenes that have not been trained in the network, especially the false alarm rate is high when the background is complex.

[0004] The Chinese patent "Training of Behavior Recognition Network, Behavior Recognition Method and Related Equipment" with patent number 2020111282295 proposes a training method for a behavior recognition network, a malicious behavior recognition method, an electronic device and a storage medium. By obtaining multiple first video frame sequences, using the behavior recognition network to extract the first feature of the first video frame sequence, using the key point detection network to extract the second feature of the first video frame sequence, based on the first feature and the second feature, the loss of the behavior recognition network is obtained and finally the behavior category is output. The disadvantage of this method is that it cannot effectively locate the location where the behavior in the video occurs.

[0005] The paper "Temporal Context Network for Activity Localization in Videos" proposes a temporal context network (TCN) for pre-localization of human activities. Similar to the Faster RCNN architecture, proposals are placed at equal intervals in videos across multiple time scales. Since pooling features within a segment alone is not sufficient to predict activity boundaries, a representation is constructed that explicitly captures the context of the proposals used to rank them. For each time segment, the features are uniformly sampled at a pair of scales and input into a temporal convolutional neural network for classification. The shortcoming of this network is that when no context is used, multiple proposals exist within or only at the boundaries of the true interval. Therefore, although the location is close to the actual interval, the boundary is not accurate. When calculating the detection metric, these nearby detections are marked as false positives, resulting in a decrease in average precision.

[0006] The paper "Asynchronous Interaction Aggregation for Action Detection" proposes an asynchronous interaction aggregation network (AIA) that uses different interactions to promote action detection. There are two key designs: one is the interaction aggregation structure (IA), which uses a unified paradigm to model and integrate multiple types of interactions; the other is the asynchronous memory update algorithm (AMU), which achieves better performance by dynamically modeling long-term interactions without huge computational costs. The disadvantage of this network is that the performance of action detection and interaction recognition is still far from ideal, and it cannot locate the location of the action in real time.

[0007] The paper "Video Action Transformer Network" proposes a Transformer-based behavioral algorithm network. This model detects and classifies actions for a certain frame in the video (this frame contains multiple characters and different actions, similar to target detection). The main idea is to borrow the transformer structure to add the information before and after the video to the final classification and positioning vector. The advantage is that the model can take into account the semantic information brought by related objects other than the target, which is very important for action recognition. The difficulty of behavior recognition and positioning usually lies in the fact that when inferring a person's behavior, it is necessary to consider not only the action of the target itself, but also the objects around it. It is equally important for inference. The disadvantage is that the model has a large amount of calculation and cannot be deployed on the edge. It can only be applied on high-performance servers. Summary of the invention

[0008] The purpose of the present invention is to address the above-mentioned shortcomings and propose a behavior recognition system that performs multi-scale prediction on large, medium and small targets in a scene and analyzes the types of behaviors that are occurring.

[0009] The present invention specifically adopts the following technical solutions:

[0010] A multi-scale prediction behavior recognition system includes a video decoding module, a video stream preprocessing module, a spatial feature extraction network and a spatiotemporal feature extraction network. The video stream preprocessing module processes the frame image processed by the video decoding module into two parts, which are respectively input into the spatial feature extraction network and the spatiotemporal feature extraction network. Specifically, the following steps are included:

[0011] Step 1: Process the video stream data of the network camera or the video stream data of the local video into a frame image through a video decoding module.

[0012] Step 2: The video stream preprocessing module processes the frame image processed by the video decoding module into two parts, which are input into the spatial feature extraction network and the spatiotemporal feature extraction network respectively.

[0013] Preferably, in step 2, the spatiotemporal feature extraction network needs to input group data, which is composed of N frames of images, N>1; the video stream preprocessing module outputs the group data, and when N=1 is set, the last frame in the group data is taken and input into the spatial feature extraction network, and when N>1 is set, the group data is input into the spatiotemporal feature extraction network to obtain time feature information; the group data N=1 is set to be represented by X, and the input image X is an RGB three-channel color image, and the following operations are performed on the data of each channel respectively, and finally the RGB three-channel image is recombined, and for channel X of image X R Pixels are calculated as follows:

[0014] (1)X R Each pixel in is converted from an integer to a decimal:

[0015]

[0016] Where W and H are the width and height of the image respectively;

[0017] (2) Image X R Each pixel in is transformed from the real domain to the logarithmic domain:

[0018] X R =log(X R ) (2)

[0019] (3) For X R The pixels in are normalized and filtered:

[0020]

[0021] X R =f i (X R ) (4)

[0022] (4) For the processed X R The result is subjected to inverse function operation, the exponent is calculated, and the logarithmic domain is transformed to the real domain:

[0023] X R =exp(X R *log(255)) (5)

[0024] (5) Repeat the above operation for the data of the G channel and the B channel to form new data:

[0025] X=(X R , X G , X B ) (6)

[0026] Assume group data N>1 and use C i Indicates that i takes values ​​from 0 to N, for group data C i Perform standardization operations on data features.

[0027] Find the mean of a group of data:

[0028]

[0029] Find the variance of group data:

[0030]

[0031] Standardize the features of the group data:

[0032]

[0033] Step 3: The feature maps output by the spatial feature extraction network and the spatiotemporal feature extraction network are fused and output as feature maps. i According to the requirements of the network, it is scaled to the specified size and input into the spatial feature extraction network and the spatiotemporal feature extraction network respectively, which specifically includes the following steps:

[0034] (1) Input X into the spatial feature extraction network to obtain feature maps of different sizes P1, P2, P3, P4, P5, ..., Pn. Feature map P1 is the data sample downsampled by 2 1 Similarly, Pn is obtained by downsampling the data sample by 2 nLet feature maps P5, P4, and P3 be represented by spatial feature vectors α, β, and λ, respectively, where α is obtained by reducing the dimension of input data X by 5 downsampling operations. The dimension of feature map X is C*H*W, C=3, H and W are the size of network input, and the output dimension of α after layer-by-layer calculation of the network is C′*H′*W′, where the value of C′ is determined by the type of behavior recognition, and H′=H / 2 5 , W′=W / 2 5 , β is the feature map obtained by downsampling the input data X four times, and the spatial dimension of β is C′*H″*W″, where the number of channels of the feature map is consistent with α, H″=H / 2 4 , W″=W / 2 4 , λ is the feature map obtained by downsampling the input data X three times, and the spatial dimension of λ is C′*H″′*W″′, where the number of channels of the feature map is consistent with α, H″′=H / 2 3 , W″′=W / 2 3 ;

[0035] (2) Group data C i Input into the spatiotemporal feature extraction network, extract features and output feature graphs C3, C4 and C5. i is four-dimensional data, Q1-*D1*H1*W1, where Q1 represents the number of channels of each image in the group data, D1 represents the length of the group data, H1 and W1 represent the height and width of the group data after data preprocessing, respectively). The spatiotemporal feature extraction network is a multi-dimensional deep network, so the dimension of the output spatiotemporal feature map is the same as the group data C. i The same, the dimension is expressed as Q2*D2*H2*W2. In order to quickly and accurately merge the features output by the spatial feature extraction network, the feature maps C3, C4 and C5 need to be input into the dimension transformation module. Taking the feature map C5 as an example, let the feature map C5 be represented by the spatiotemporal feature vector γ. The module processes the feature map as follows:

[0036] 1) The spatiotemporal feature map is input into the spatiotemporal convolution kernel, reducing the second dimension of the feature map to 1

[0037] γ=f conv (γ), (γ∈C1*1*H1*W1)

[0038] 2) Input the feature map γ into the dimension compression unit, f squeeze ( ) is a dimension compression function that reduces 4-dimensional data to 3-dimensional data.

[0039]

[0040] 3) The feature map γ is scaled to increase the size of the width and height dimensions through the scale expansion unit, so that the width and height of γ are equal to those of the feature map C4. The specific operations are as follows:

[0041] (1) Set appropriate parameters to make the cubic polynomial F(x) infinitely close to the best interpolation function sin(x) / x;

[0042] (2) The pixel value to be inserted after the scale is set to be enlarged and represented by (x, y), (i, j) is the reference pixel coordinate of the pixel to be inserted, and the new pixel value is calculated as follows:

[0043] f(x,y)=f(i+u,j+v)=ABC(0<u<1,0<v<1) (12)

[0044] in:

[0045]

[0046] According to the above steps, each pixel in the feature map γ is calculated to generate γ', and the output feature map is fused with the feature maps corresponding to C3 and C4.

[0047] The scale prediction for the target includes small-scale prediction for large targets and intermediate-scale prediction for medium targets. The small-scale prediction process for large targets is as follows:

[0048] The spatiotemporal feature map γ' is input into the dynamic learning module, which includes a dynamic learning unit and a feature decoding unit. The dynamic feature fusion unit is as follows: Figure 2 As shown, the dynamic learning unit operates the spatiotemporal feature map γ' according to the following steps:

[0049] 1) First, the spatiotemporal feature vector γ'γ′∈C1*H1*W1, the width and height dimensions are consistent with α, that is, H1=H′, W1-W′. The feature map γ′ is filtered by the weighted convolution kernel m to output a new weight vector I, which is calculated as follows:

[0050] I=f 3*3 (γ′), (I∈C1*H1*W1)

[0051] 2) The weight vector I is characterized and then output as a weight matrix T, which combines the width and height dimensions into a one-dimensional vector to output the weight:

[0052] T=f reshape (l),T∈C1*D, where the size of D is H1*W1

[0053] 3) Transpose the weight matrix to generate the transposed weight matrix T′:

[0054] T′=T T

[0055] 4) Multiply the matrix T by the matrix T' to generate a new weight matrix of different dimensions, input the matrix into the sigmoid function, and output the weight matrix Q

[0056] Q=Sigmoid(T.T'),(Q∈C1*C1)

[0057] 5) Multiply the weight matrix Q by the feature map T to generate the weight matrix Q'(Q'∈C1*D)

[0058] 6) The feature map γ′ is converted into a feature matrix E(E∈C1*D) by formula (12)

[0059] 7) After adding the feature matrix E and the weight matrix Q', the reshape operation is performed according to formula 11 to generate the dynamically fused spatiotemporal feature map F∈C1*H1*W1

[0060] F=f reshape (E+Q′)

[0061] Add the spatiotemporal feature map F to the spatial feature map α to perform the feature fusion operation of spatial features and spatiotemporal features:

[0062]

[0063] Feature map O1 is the feature map after the feature fusion of the spatiotemporal feature extraction network and the spatial feature extraction network, namely, P5 and C5. This feature map contains the category information of the action and the size and position information of the target.

[0064] According to the above steps, the P4 of the spatial feature extraction network and the C4 of the spatiotemporal feature extraction network are dynamically fused to output the feature map O2, completing the medium target action classification and size position encoding; the P3 of the spatial feature extraction network and the C3 of the spatiotemporal feature extraction network are dynamically fused to output the feature map O3, completing the small target action classification and size position encoding. The spatial feature extraction network used in this patent only uses three scales of spatial feature prediction. The output number can be appropriately reduced or increased according to the hardware requirements and the actual use environment. The spatial feature extraction network and the spatiotemporal feature extraction network can also be replaced with a basic network model with better accuracy according to the actual use environment.

[0065] Step 4: Decode the output feature map. The decoding unit exists in the dynamic learning module. The specific operation process is as follows:

[0066] Taking into account the diversity of behaviors, in actual scene applications, there are usually multiple behaviors of a single target. For example, body movements such as raising hands may occur during walking, and multiple behaviors often appear at the same time. Therefore, when the feature fusion feature maps O1, O2, and O3 feature maps are decoded for action classification, the activation unit is a sigmoid function, which is specifically used in the multi-classification algorithm of behavior. The calculation method is as follows:

[0067]

[0068] In order to adapt to the categories in the current data set during training, the following loss function is used for the category prediction model, and its loss value is loss c The calculation formula is:

[0069] loss object =-Σb′*Lnb (17)

[0070] Where b′ represents the true value in the label, and b represents the category output value predicted by the model;

[0071] Position information loss function loss value loss coord Calculation formula:

[0072] loss coord =-y′*log(y)-(1-y′)*log(1-y) (18)

[0073] Where y′ represents the true coordinate value in the label, and y represents the output value of the model predicted coordinate.

[0074] The present invention has the following beneficial effects:

[0075] The multi-scale prediction behavior recognition system performs multiple predictions at different scales on the same input video clip, and analyzes the types of behaviors that are occurring, solving the problems of missed detection and false detection in actual complex scenes. Multi-scale prediction improves the utilization of image features extracted by the network and increases the accuracy of recognition. At the same time, a new calculation method is used in action prediction to achieve multi-label prediction, which reduces the false alarm rate and missed alarm rate, and improves the generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A block diagram of a multi-scale prediction behavior recognition system;

[0077] Figure 2 Schematic diagram of the dynamic feature fusion unit. DETAILED DESCRIPTION

[0078] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0079] Combination Figure 1 A multi-scale prediction behavior recognition system includes a video decoding module, a video stream preprocessing module, a spatial feature extraction network and a spatiotemporal feature extraction network. The video stream preprocessing module processes the frame image processed by the video decoding module into two parts, which are input into the spatial feature extraction network and the spatiotemporal feature extraction network respectively. Specifically, the following steps are included:

[0080] Step 1, processing the video stream data of the network camera or the video stream data of the local video into a frame image through a video decoding module;

[0081] Step 2, the video stream preprocessing module processes the frame image processed by the video decoding module into two parts, which are input into the spatial feature extraction network and the spatiotemporal feature extraction network respectively;

[0082] Step 3: Input the data samples into the spatial feature extraction network and the spatiotemporal feature extraction network respectively; perform fusion operation on the feature maps output by the spatial feature extraction network and the spatiotemporal feature extraction network respectively to output the feature maps.

[0083] Step 4: Decode the output feature map.

[0084] The spatiotemporal feature extraction network needs to input group data, which is composed of N frames of images, N>1; the video stream preprocessing module outputs group data, and when N=1, the last frame in the group data is taken and input into the spatial feature extraction network. When N>1, the group data is input into the spatiotemporal feature extraction network to obtain temporal feature information; the group data N=1 is represented by X, and the input image X is an RGB three-channel color image. The following operations are performed on the data of each channel respectively, and finally the RGB three-channel image is recombined. For channel X of image X R Pixels are calculated as follows:

[0085] (1)X R Each pixel in is converted from an integer to a decimal:

[0086]

[0087] Where W and H are the width and height of the image respectively;

[0088] (2) Image X R Each pixel in is transformed from the real domain to the logarithmic domain:

[0089] X R =log(X R ) (2)

[0090] (3) For X R The pixels in are normalized and filtered:

[0091]

[0092] X R =f l (X R ) (4)

[0093] (4) For the processed X R The result is subjected to inverse function operation, the exponent is calculated, and the logarithmic domain is transformed to the real domain:

[0094] X R =exp(X R *log(255)) (5)

[0095] (5) Repeat the above operation for the data of the G channel and the B channel to form new data:

[0096] X=(X R , X G , X B ) (6)

[0097] Assume group data N>1 and use C i Indicates that i takes values ​​from 0 to N, for group data C i Perform standardization operations on data features.

[0098] Find the mean of a group of data:

[0099]

[0100] Find the variance of group data:

[0101]

[0102] Standardize the features of the group data:

[0103]

[0104] The data samples X and C i According to the requirements of the network, it is scaled to the specified size and input into the spatial feature extraction network and the spatiotemporal feature extraction network respectively, which specifically includes the following steps:

[0105] (1) After X is input into the spatial feature extraction network, the three-layer output spatial feature vectors α, β and λ are obtained, where α is obtained by reducing the dimension of the input data X by 5 downsampling operations. The dimension of the feature map X is C*H*W, C=3, H and W are the size of the network input respectively. After the α output is calculated layer by layer, the output dimension is C′*H′*W′, where the value of C′ is determined by the type of behavior recognition, and H′=H / 2 5 , W′=W / 25 , β is the feature map obtained by downsampling the input data X four times, and the spatial dimension of β is C′*H″*W″, where the number of channels of the feature map is consistent with α, H″=H / 2 4 , W″=W / 2 4 , λ is the feature map obtained by downsampling the input data X three times, and the spatial dimension of λ is C′*H″′*W″′, where the number of channels of the feature map is consistent with α, H″′=H / 2 3 , W″′=W / 2 3 ;

[0106] (2) Group data C i Input into the spatiotemporal feature extraction network to obtain the spatiotemporal feature vector γ. Since the spatiotemporal feature extraction network inputs group data C i is four-dimensional data, Q1*D1*H1*W1, where Q1 represents the number of channels of each image in the group data, D1 represents the length of the group data, H1 and W1 represent the height and width of the group data after data preprocessing, respectively). The spatiotemporal feature extraction network is a multi-dimensional deep network, so the dimension of the output γ is the same as C i Similarly, the dimension of γ is expressed as Q2*D2*H2*W2. In order to quickly and accurately merge the features output by the spatial feature extraction network, the feature map output by the spatiotemporal feature extraction network must be dimensionally compressed:

[0107]

[0108] f squeeze () is a dimension compression function. The relevant hyperparameters are set in the spatiotemporal feature extraction network so that the value of the second dimension of the final output γ is 1.

[0109] The scale prediction for the target includes small-scale prediction for large targets and intermediate-scale prediction for medium targets. The small-scale prediction process for large targets is as follows:

[0110] γ is also C i Each image in is downsampled 5 times, so the width and height dimensions are the same as the spatial feature vector α. Add the compressed dimension γ to α:

[0111]

[0112] Feature map O1 is the feature map after the small-size output of the spatiotemporal feature extraction network and the spatial feature extraction network, i.e., α and γ, are fused. This feature map contains the category information of the action and the size and position information of the target.

[0113] The intermediate scale prediction process for medium targets is:

[0114] Since the width and height dimensions of the feature γ output by the spatiotemporal feature extraction network are the same as α, although it has been processed by the dimension compression of formula (10), the height and width dimensions of the feature map are still half of the feature map β. Therefore, to complete the prediction of the intermediate scale, it is necessary to first scale up the width and height dimensions of γ processed in formula (10). The operation steps are as follows:

[0115] (1) Set appropriate parameters to make the cubic polynomial F(x) infinitely close to the best interpolation function sin(x) / x;

[0116] (2) The pixel value to be inserted after the scale is set to be enlarged and represented by (x, y), (i, j) is the reference pixel coordinate of the pixel to be inserted, and the new pixel value is calculated as follows:

[0117] f(x,y)=f(i+u,j+v)=ABC(0<u<1,0<v<1) (12)

[0118] in:

[0119]

[0120] According to the above steps, each pixel in the feature map γ is calculated, and the result γ′ is added to the feature map β output by the spatial feature extraction network according to formula 11 to output the feature map O2, thereby completing the medium target action classification and size position encoding; γ′ is input to the next scale expansion unit, and the output result γ″ is calculated according to the above operation, and the feature map λ output by the spatial feature extraction network is added according to formula 11 to output the feature map O3, thereby completing the small target action classification and size position encoding.

[0121] The spatiotemporal feature map γ' is input into the dynamic learning module, which includes a dynamic learning unit and a feature decoding unit. The dynamic feature fusion unit is as follows: Figure 2 As shown, the dynamic learning unit operates the spatiotemporal feature map γ' according to the following steps:

[0122] 1) First, the spatiotemporal feature vector γ'γ′∈C1*H1*W1, the width and height dimensions are consistent with α, that is, H1=H′, W1=W′. The feature map γ′ is filtered by the weighted convolution kernel m to output a new weight vector I, which is calculated as follows:

[0123] I=f 3*3 (γ′), (I∈C1*H1*W1)

[0124] 2) The weight vector I is characterized and then output as a weight matrix T, which combines the width and height dimensions into a one-dimensional vector to output the weight:

[0125] T=f reshape (I),T∈C1*D, where the size of D is H1*W1

[0126] 3) Transpose the weight matrix to generate the transposed weight matrix T′:

[0127] T′=T T

[0128] 4) Multiply the matrix T by the matrix T' to generate a new weight matrix of different dimensions, input the matrix into the sigmoid function, and output the weight matrix Q

[0129] Q=Sigmoid(T﹒T'),(Q∈C1*C1)

[0130] 5) Multiply the weight matrix Q by the feature map T to generate the weight matrix Q'(Q'∈C1*D)

[0131] 6) The feature map γ′ is converted into a feature matrix E(E∈C1*D) by formula (12)

[0132] 7) After adding the feature matrix E and the weight matrix Q', the reshape operation is performed according to formula 11 to generate the dynamically fused spatiotemporal feature map F∈C1*H1*W1

[0133] F=f reshape (E+Q′)

[0134] Add the spatiotemporal feature map F to the spatial feature map α to perform the feature fusion operation of spatial features and spatiotemporal features:

[0135]

[0136] Feature map O1 is the feature map after feature fusion of the small-size output of the spatiotemporal feature extraction network and the spatial feature extraction network, namely P5 and C5. This feature map contains the category information of the action and the size and position information of the target.

[0137] According to the above steps, the P4 of the spatial feature extraction network and the C4 of the spatiotemporal feature extraction network are dynamically fused to output the feature map O2, completing the medium target action classification and size position encoding; the P3 of the spatial feature extraction network and the C3 of the spatiotemporal feature extraction network are dynamically fused to output the feature map O3, completing the small target action classification and size position encoding. The spatial feature extraction network used in this patent only uses three scales of spatial feature prediction. The output number can be appropriately reduced or increased according to the hardware requirements and the actual use environment. The spatial feature extraction network and the spatiotemporal feature extraction network can also be replaced with a basic network model with better accuracy according to the actual use environment.

[0138] The specific process of step 4 is:

[0139] Considering the diversity of behaviors, in actual scene applications, there are usually multiple behaviors of a single target. For example, body movements such as raising hands may occur during walking. Multiple behaviors often occur at the same time. Therefore, when decoding the action classification of feature fusion vectors O1, O2, and O3, the activation unit is a sigmoid function. This function is specifically used in the multi-classification algorithm of behavior. The calculation method is as follows:

[0140]

[0141] In order to adapt to the categories in the current data set during training, the following loss function is used for the category prediction model, and its loss value is loss c The calculation formula is:

[0142] loss object =-∑b′*Lnb (17)

[0143] Where b′ represents the true value in the label, and b represents the category output value predicted by the model;

[0144] Position information loss function loss value loss coord Calculation formula:

[0145] loss coord =-y′*log(y)-(1-y)*log(1-y) (18)

[0146] Where y′ represents the true coordinate value in the label, and y represents the output value of the model predicted coordinate.

[0147] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A multi-scale prediction behavior recognition system, characterized in that: It includes a video decoding module, a video stream preprocessing module, a spatial feature extraction network and a spatiotemporal feature extraction network. The video stream preprocessing module processes the frame image processed by the video decoding module into two parts, which are input into the spatial feature extraction network and the spatiotemporal feature extraction network respectively. Specifically, it includes the following steps: Step 1, processing the video stream data of the network camera or the video stream data of the local video into a frame image through a video decoding module; Step 2: The video stream preprocessing module processes the frame image processed by the video decoding module into two parts, which are input into the spatial feature extraction network and the spatiotemporal feature extraction network respectively; Step 3, fusing the feature maps output by the spatial feature extraction network and the spatiotemporal feature extraction network to output feature maps; Step 4, decoding the output feature map; In step 2, the spatiotemporal feature extraction network needs to input group data, which is composed of N frames of images, N>1; the video stream preprocessing module outputs group data, and when N=1, the last frame in the group data is taken and input into the spatial feature extraction network. When N>1, the group data is input into the spatiotemporal feature extraction network to obtain temporal feature information; the group data N=1 is represented by X, and the input image X is an RGB three-channel color image. The following operations are performed on the data of each channel respectively, and finally the RGB three-channel image is recombined. For channel X of image X R Pixels are calculated as follows: (1)X R Each pixel in is converted from an integer to a decimal: Where W and H are the width and height of the image respectively; (2) Image X R Each pixel in is transformed from the real domain to the logarithmic domain: X R =log(X R ) (2) (3) For X R The pixels in are normalized and filtered: X R =f l (X R ) (4) (4) For the processed X R The result is subjected to inverse function operation, the exponent is calculated, and the logarithmic domain is transformed to the real domain: X R =exp(X R *log(255)) (5) (5) Repeat the above operation for the data of the G channel and the B channel to form new data: X=(X R ,X G ,X B ) (6) Assume group data N>1 and use C i Indicates that i takes values ​​from 0 to N, for group data C i Perform standardization operations on data features. Find the mean of a group of data: Find the variance of group data: Standardize the features of the group data:

2. The multi-scale prediction behavior recognition system according to claim 1, characterized in that: In step 3, the data samples X and Ci are scaled to the specified size according to the requirements of the network and input into the spatial feature extraction network and the spatiotemporal feature extraction network respectively, which specifically includes the following steps: (1) After X is input into the spatial feature extraction network, a three-layer output spatial feature vector α, β and λ are obtained, where α is obtained by reducing the dimension of the input data X by 5 downsampling operations. The dimension of the feature map X is C*H*W, C=3, H and W are the sizes of the network input respectively. After the α output is calculated layer by layer, the output dimension is C′*H′*W′, where the value of C′ is determined by the type of behavior recognition, H′=H / 25, W′=W / 25, β is the feature map obtained by downsampling the input data X by 4 times, the spatial dimension of β is C′*H″*W″, where the number of channels of the feature map is consistent with α, H″=H / 24, W″=W / 24, α is the feature map obtained by downsampling the input data X by 3 times, the spatial dimension of λ is C′*H″′*W″′, where the number of channels of the feature map is consistent with α, H″′=H / 23, W″′=W / 23; (2) Group data C i Input into the spatiotemporal feature extraction network to obtain the spatiotemporal feature vector γ. Since the spatiotemporal feature extraction network inputs group data C i is four-dimensional data, Q1*D1*H1*W1, where Q1 represents the number of channels of each image in the group data, D1 represents the length of the group data, H1 and W1 represent the height and width of the group data after data preprocessing, respectively). The spatiotemporal feature extraction network is a multi-dimensional deep network, so the dimension of the output γ is the same as the group data C. i Similarly, the dimension of γ is expressed as Q2*D2*H2*W2. In order to quickly and accurately merge the features output by the spatial feature extraction network, the feature map output by the spatiotemporal feature extraction network must be dimensionally compressed: f squeeze () is a dimension compression function. The relevant hyperparameters are set in the spatiotemporal feature extraction network so that the value of the second dimension of the final output γ is 1.

3. A multi-scale prediction behavior recognition system as claimed in claim 2, characterized in that: The multi-scale prediction behavior recognition system according to claim 1 is characterized in that the scale prediction for the target includes a small-scale prediction for a large target and an intermediate-scale prediction for a medium target, and the small-scale prediction process for the large target is: The space-time eigenvector γ is also the C i Each image in is downsampled 5 times, so the width and height dimensions are the same as α. Add the compressed γ to α: Feature map O1 is the feature map after the feature fusion of γ and α, which is the small-size output of the spatiotemporal feature extraction network and the spatial feature extraction network. This feature map contains the category information of the action and the size and position information of the target. The intermediate scale prediction process for medium targets is: Since the width and height dimensions of the feature γ output by the spatiotemporal feature extraction network are the same as α, although it has been processed by the dimension compression of formula (10), the height and width dimensions of the feature map are still half of the feature map β. Therefore, to complete the prediction of the intermediate scale, it is necessary to first scale up the width and height dimensions of γ processed in formula (10). The operation steps are as follows: (1) Set appropriate parameters to make the cubic polynomial F(x) infinitely close to the best interpolation function sin(x) / x; (2) The pixel value to be inserted after the scale is set to be enlarged and represented by (x, y), (i, j) is the reference pixel coordinate of the pixel to be inserted, and the new pixel value is calculated as follows: f(x,y)=f(i+u,j+v)=ABC(0<u<1,0<v<1) (12) in: According to the above steps, each pixel in the feature map γ is calculated, and the result γ′ is added to the feature map β output by the spatial feature extraction network according to formula 11 to output the feature map O2, thereby completing the medium target action classification and size position encoding; γ′ is input to the next scale expansion unit, and the output result γ″ is calculated according to the above operation, and the feature map λ output by the spatial feature extraction network is added according to formula 11 to output the feature map O3, thereby completing the small target action classification and size position encoding.

4. The multi-scale prediction behavior recognition system according to claim 1, characterized in that: The specific process of step 4 is: When the feature fusion vector O1, O2, and O3 feature maps are decoded for action classification, the activation unit used is the sigmoid function, which is specifically used in the multi-classification algorithm of behavior. The calculation method is as follows: In order to adapt to the categories in the current data set during training, the following loss function is used for the category prediction model, and its loss value is loss c The calculation formula is: loss object =-∑b′*lnb (17) Where b′ represents the true value in the label, and b represents the category output value predicted by the model; Position information loss function loss value loss coord Calculation formula: loss coord =-y′*log(y)-(1-y′)*log(1-y) (18) Where y′ represents the true coordinate value in the label, and y represents the output value of the model predicted coordinate.

Images