Behavior Detection Method and Device Based on Embedding an Arbitrary Boundary Matching Nomination Network

Abstract

This application relates to a behavior detection method and device based on an embedded arbitrary boundary matching nomination network. The method includes: calling a backbone network to extract features from an unclipped video to obtain a first feature; calling an encoder-decoder module to encode and decode the first feature to obtain a second feature; calling an embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain multiple ordered combined probabilities; calling a behavior classifier to identify the multiple ordered combined probabilities to obtain a behavior detection result. The method simultaneously solves the behavior boundary prediction problem and the behavior classification problem end-to-end by calling the encoder-decoder module; supplements the single-stage behavior detection result by calling the embedded boundary matching nomination network module, improving the recall rate of positive samples; reduces boundary prediction noise by calling a cascaded boundary refinement module, enhancing the behavior detection accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a behavior detection method and device based on an embedded arbitrary boundary matching nomination network. Background Art

[0002] Existing multi-stage behavior detection methods generally include the behavior temporal position nomination stage, the boundary optimization stage of the nominated behavior temporal position, the behavior classification stage, and the redundant box suppression stage, etc. Since each stage requires one or more models to implement, it leads to low efficiency.

[0003] Existing single-stage behavior detection methods generally share a common feature and jointly model both the behavior location prediction problem and the behavior type problem based on this feature. This approach addresses the efficiency issues of multi-stage behavior detection algorithm frameworks by using a "one feature, multiple problems" approach. However, single-stage behavior detection methods are often affected by factors such as the number of model parameters and data volume, ultimately resulting in their accuracy failing to reach the level of multi-stage behavior detection methods. Summary of the Invention

[0004] Based on the above technical problems, in order to solve the low efficiency problem of existing multi-stage behavior detection methods and single-stage behavior detection methods, the present invention proposes a new behavior detection method based on the encoder-decoder module of behavior detection using the Transformer neural network structure and the embedded boundary matching nomination network to improve the accuracy of end-to-end behavior detection.

[0005] A first aspect of the present invention provides a behavior detection method based on an embedded arbitrary boundary matching nomination network, wherein behavior detection is performed using a behavior detection device based on an embedded arbitrary boundary matching nomination network, wherein the behavior detection device based on an embedded arbitrary boundary matching nomination network includes a backbone network, an encoder-decoder module, an embedded boundary matching nomination network module, and a behavior classifier; the method includes:

[0006] Call the backbone network to extract features from the unedited video and obtain the first feature;

[0007] Calling the encoder-decoder module to encode and decode the first feature to obtain a second feature;

[0008] Calling the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain multiple ordered combination probabilities;

[0009] A behavior classifier is called to identify the multiple ordered combination probabilities to obtain a behavior detection result.

[0010] Specifically, the encoder-decoder module consists of an encoding module and a decoding module; the calling of the encoder-decoder module to encode and decode the first feature to obtain a second feature includes:

[0011] Converting the first feature into a temporal feature with temporal position encoding information through the encoding module;

[0012] Constructing interdependencies between features at different temporal positions through the encoder to obtain temporal high-level semantic features;

[0013] Extracting corresponding temporal position information and behavior information from the temporal high-level semantic features through the decoder as the second feature.

[0014] Specifically, both the encoding module and the decoding module include a local temporal self-attention module and a global temporal self-attention module, and the decoding module further includes a local masked self-attention module and a global masked self-attention module.

[0015] Furthermore, the calling of the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain multiple ordered combination probabilities includes:

[0016] Inputting the second feature into the embedded boundary matching nomination network module to obtain a segment start point prediction probability, a segment end point prediction probability, and a boundary matching confidence map;

[0017] Calculating the start and end joint probabilities that meet the preset constraint conditions based on the segment start point prediction probability, the segment end point prediction probability, and the boundary matching confidence map, and obtaining multiple combination probabilities;

[0018] Sorting the multiple combination probabilities in descending order to obtain multiple ordered combination probabilities.

[0019] Furthermore, the behavior detection device based on the embedded arbitrary boundary matching nomination network further includes a cascaded boundary refinement module. Before inputting the multiple ordered combination probabilities into the behavior classifier to obtain the behavior detection result, it further includes: inputting the multiple ordered combination probabilities into the cascaded boundary refinement module for refinement operations.

[0020] Still further, the inputting of the multiple ordered combination probabilities into the cascaded boundary refinement module for refinement operations includes:

[0021] Obtaining a start point prediction probability matrix and a segment end point prediction probability matrix;

[0022] Performing a dot product of the start point prediction probability matrix and the transposed matrix of the segment end point prediction probability matrix to obtain a first combined confidence map;

[0023] Perform a Hadamard product operation on the first combined confidence map, a full-ones upper triangular matrix, and a full-ones matrix with a diagonal of 0 to obtain a second combined confidence map;

[0024] Based on the calculation and zero-padding operation of the first combined confidence map, the second combined confidence map, and the boundary matching confidence map, obtain a third combined confidence map and use it as the refined boundary result.

[0025] Preferably, the behavior classifier includes a behavior classifier based on a fuzzy neural network and a behavior classifier applicable to nomination segments.

[0026] A second aspect of the present invention provides a behavior detection device based on an embedded arbitrary boundary matching nomination network, the device comprising:

[0027] A backbone network for extracting features from an unclipped video to obtain a first feature;

[0028] An encoder-decoder module for encoding and decoding the first feature to obtain a second feature;

[0029] An embedded boundary matching nomination network module for performing boundary matching nomination processing on the second feature to obtain a plurality of ordered combined probabilities;

[0030] A cascaded boundary refinement module for refining the plurality of ordered combined probabilities to obtain a refined boundary result;

[0031] A behavior classifier for identifying the refined boundary result to obtain a behavior detection result.

[0032] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0033] Invoke the backbone network to extract features from an unclipped video to obtain a first feature;

[0034] Invoke the encoder-decoder module to encode and decode the first feature to obtain a second feature;

[0035] Invoke the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain a plurality of ordered combined probabilities;

[0036] Invoke the behavior classifier to identify the plurality of ordered combined probabilities to obtain a behavior detection result.

[0037] A fourth aspect of the present invention provides a computer program product, including a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0038] Call the backbone network to extract features from the unedited video and obtain the first feature;

[0039] Calling the encoder-decoder module to encode and decode the first feature to obtain a second feature;

[0040] Calling the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain multiple ordered combination probabilities;

[0041] A behavior classifier is called to identify the multiple ordered combination probabilities to obtain a behavior detection result.

[0042] The beneficial effects of the present application are as follows: the method described in the present application solves the behavior boundary prediction problem and the behavior classification problem in an end-to-end manner by calling the encoder-decoder module including the local temporal self-attention module and the global temporal self-attention module; by calling the embedded boundary matching nomination network module, boundary matching nomination processing is performed to obtain multiple ordered combination probabilities, and the nomination network supplements the single-stage behavior detection results and improves the recall rate of positive samples; by calling the cascaded boundary refinement module, the multiple ordered combination probabilities are refined to obtain more refined segment boundary results, reduce boundary prediction noise, and improve the accuracy of behavior detection and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0044] The present application can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0045] Figure 1 A schematic diagram of the method steps of an exemplary embodiment of the present application is shown;

[0046] Figure 2 A schematic diagram of the encoder-decoder structure of an exemplary embodiment of the present application is shown;

[0047] Figure 3 A schematic diagram of a method process in an exemplary embodiment of the present application is shown;

[0048] Figure 4 A schematic diagram of the device structure of an exemplary embodiment of the present application is shown;

[0049] Figure 5 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application is shown;

[0050] Figure 6 A schematic diagram of a storage medium provided by an exemplary embodiment of the present application is shown. Detailed Implementation Modes

[0051] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present application. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application. It is obvious to those skilled in the art that the present application can be implemented without one or more of these details. In other examples, some technical features well known in the art are not described to avoid confusion with the present application.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or combinations thereof.

[0053] Now, exemplary embodiments according to the present application will be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many different forms and should not be construed as being limited only to the embodiments set forth herein. The accompanying drawings are not drawn to scale, and some details may be enlarged for the purpose of clear expression, and some details may be omitted. The shapes of various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0054] The following is combined with the specification appendix Figure 1-6 Several embodiments are given to describe the exemplary implementation manners according to the present application. It should be noted that the following application scenarios are only shown for the convenience of understanding the spirit and principle of the present application, and the implementation manners of the present application are not limited in this regard. On the contrary, the implementation manners of the present application can be applied to any applicable scenario.

[0055] Embodiment 1:

[0056] This embodiment implements a behavior detection method based on an embedded arbitrary boundary matching nomination network, and uses a behavior detection device based on the embedded arbitrary boundary matching nomination network for behavior detection. Among them, the behavior detection device based on the embedded arbitrary boundary matching nomination network includes a backbone network, an encoder-decoder module, an embedded boundary matching nomination network module, and a behavior classifier; asFigure 1 As shown, the method includes:

[0057] S1. Invoke the backbone network to extract features from the unclipped video, obtaining the first feature;

[0058] S2. Invoke the encoder-decoder module to encode and decode the first feature, obtaining the second feature;

[0059] S3. Invoke the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature, obtaining multiple ordered combined probabilities;

[0060] S4. Invoke the behavior classifier to identify the multiple ordered combined probabilities, obtaining the behavior detection result.

[0061] Specifically, the encoder-decoder module consists of an encoding module and a decoding module; invoking the encoder-decoder module to encode and decode the first feature to obtain the second feature includes: converting the first feature into a temporal feature with temporal position encoding information through the encoding module; constructing an interdependent relationship between features at different temporal positions through the encoder to obtain a temporal high-level semantic feature; extracting corresponding temporal position information and behavior information from the temporal high-level semantic feature through the decoder as the second feature.

[0062] Specifically, both the encoding module and the decoding module include a local temporal self-attention module and a global temporal self-attention module, and the decoding module further includes a local masked self-attention module and a global masked self-attention module.

[0063] Further, invoking the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain multiple ordered combined probabilities includes: inputting the second feature into the embedded boundary matching nomination network module to obtain the segment start point prediction probability, the segment end point prediction probability, and the boundary matching confidence map; calculating the start and end joint probabilities that meet the preset constraint conditions based on the segment start point prediction probability, the segment end point prediction probability, and the boundary matching confidence map and obtaining multiple combined probabilities; sorting the multiple combined probabilities in descending order to obtain multiple ordered combined probabilities.

[0064] Further, the behavior detection device based on the embedded arbitrary boundary matching nomination network further includes a cascaded boundary refinement module. Before inputting the multiple ordered combined probabilities into the behavior classifier to obtain the behavior detection result, it further includes: inputting the multiple ordered combined probabilities into the cascaded boundary refinement module for refinement operations.

[0065] Further, input multiple ordered combined probabilities into a cascaded boundary refinement module for refinement operations, including: obtaining a starting point prediction probability matrix and an ending point prediction probability matrix of a segment; performing a dot product on the starting point prediction probability matrix and the transposed matrix of the ending point prediction probability matrix of the segment to obtain a first combined confidence map; performing a Hadamard product operation on the first combined confidence map, a full 1 upper triangular matrix, and a full 1 matrix with a diagonal of 0 to obtain a second combined confidence map; based on the calculation and zero-padding operation of the first combined confidence map, the second combined confidence map, and a boundary matching confidence map, obtaining a third combined confidence map and using it as a refined boundary result.

[0066] Preferably, the behavior classifier includes a behavior classifier based on a fuzzy neural network and a behavior classifier applicable to nominated segments.

[0067] The method described in this application simultaneously solves the behavior boundary prediction problem and the behavior classification problem end-to-end by calling an encoder-decoder module that includes a local temporal self-attention module and a global temporal self-attention module; by calling an embedded boundary matching nomination network module to perform boundary matching nomination processing to obtain multiple ordered combined probabilities, the nomination network supplements the single-stage behavior detection result and improves the recall rate of positive samples; by calling a cascaded boundary refinement module to perform refinement operations on the multiple ordered combined probabilities, a more refined segment boundary result is obtained, reducing boundary prediction noise and improving the accuracy of behavior detection and recognition.

[0068] Embodiment 2:

[0069] This embodiment implements a behavior detection method based on an embedded arbitrary boundary matching nomination network, and uses a behavior detection device based on an embedded arbitrary boundary matching nomination network to perform behavior detection. Among them, the behavior detection device based on an embedded arbitrary boundary matching nomination network includes a backbone network, an encoder-decoder module, an embedded boundary matching nomination network module, and a behavior classifier; the method includes: calling the backbone network to extract features from an unclipped video to obtain a first feature; calling the encoder-decoder module to encode and decode the first feature to obtain a second feature; calling the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain multiple ordered combined probabilities; calling the behavior classifier to identify the multiple ordered combined probabilities to obtain a behavior detection result.

[0070] In a possible specific implementation, call the backbone network to extract features from an unclipped video. The backbone network uses an I3D network (Inflated 3D ConvNet, i.e., inflated convolutional network) with a two-stream structure to extract features from the unclipped video. For example, given an uncropped video with a temporal length (in frames) of l v of the uncropped video Sampling is performed in small segments with σ=16 frames, and 16 frames of RGB image I rgb ∈R 3×16 Or optical flow map I opt ∈R 1×16 As input to the I3D network, the RGB-based sequence features are obtained respectively. and sequence features based on optical flow graphs Among them, 2048 is the feature channel dimension value, l f is the length of the sequence feature, i.e. l f =l v / σ, and finally F opt Splicing in F in the feature channel direction rgb Get later The above steps can be expressed by the following mathematical formula:

[0071] F rgb =I3D(I rgb )

[0072] F opt =I3D(I opt )

[0073] F seq =Concat(F rgb , F opt ) c

[0074] Among them, Concat means concatenation operation in the feature channel direction. Based on the above feature extraction operation, the length of the temporal features obtained for different video lengths is also different. Since the mini-batch strategy is used during training, usually a batch is used as input, we need to perform the above temporal feature F seq Sampling is performed to uniformly obtain time series features with a time series length of 100

[0075] This is the first feature mentioned above.

[0076] As a variable implementation, in order to make the timing basis feature In order to avoid losing the temporal position information after being encoded by the encoder, this invention adds a learnable position encoding module (LPE, Learnable Position Embedding). Before training, this module uses the Xavier Uniform method to embed the vector V∈R 4096 ×100 During training, each time before entering the encoder-decoder module, Get the temporal feature F′ with temporal position encoding information seq ∈R 4096×100. In practice, compared with the absolute position encoding or relative position encoding schemes with fixed parameters used in TadTR and other solutions, the results show that the LPE has better effects. The reason is that since V is trainable, it will continuously update the parameters in V according to the current performance of the model during the model training process until it finds more suitable parameters for expressing the temporal position information to fit the data feature distribution.

[0077] Further, the encoder-decoder module is called to encode and decode the first feature to obtain a second feature. Here, the encoder-decoder module includes an encoding module and a decoding module. Both the encoding module and the decoding module include a local temporal self-attention module and a global temporal self-attention module. The decoding module further includes a local masked self-attention module and a global masked self-attention module.

[0078] In a possible specific implementation, the encoding module consists of a local temporal self-attention module (Local Self-Attention, hereinafter referred to as LSA), a global temporal self-attention module (Global Self-Attention, hereinafter referred to as GSA), layer normalization (Layer Normalization, hereinafter referred to as LN), and a fully connected feed-forward network (Full Feedforward Network, hereinafter referred to as FFN). Both the local temporal self-attention module and the global temporal self-attention module adopt the multi-head self-attention module mechanism in the Transformer decoder, but the local temporal self-attention module is different and adopts a local window calculation method. For example, after obtaining F′ seq , we need to use three linear transformations to obtain three tensors of Query, Key, and Value as the input to the encoder. The specific calculation process can be expressed by the following mathematical formula:

[0079] Query = γ(F′ seq )

[0080] Key = δ(F′ seq )

[0081] Value = p(F′ seq )

[0082] E loc = LSA(Query, Key, Value|winsize)

[0083] E global = GSA(Query, Key, Value)

[0084] E fusion = Concat(Eloc , E global ) c ·W o

[0085] E out = LN(F' seq + FFN(LN(F' seq + E fusion )))

[0086] where winslze is the size of the local window, and W o is a learnable parameter matrix for loc and E global information fusion. E loc and E global The calculation processes of and E are parallel and completed simultaneously. The global temporal self-attention module is responsible for capturing the global information of long videos, but its disadvantage is introducing global noise and being insensitive to the boundaries of small target segments. Therefore, a local temporal self-attention module is introduced to solve the small target segment problem and be responsible for capturing local temporal information. The calculation process of the global temporal self-attention module (GSA) can be expressed by the following mathematical form:

[0087]

[0088] The mathematical expression of the local temporal self-attention module (LSA) is:

[0089]

[0090] The above q, k, and v come from Qusry, Key, and Value respectively.

[0091] The decoding module is structurally corresponding to the encoding module, and its mechanism is similar to that of the encoding module, both using the self-attention mechanism. The difference is that: the decoding module uses the output of the previous decoding module to construct Query, while Key and Value come from the final output of the encoder. In addition to LSA and GSA, the decoding module also has a local masked self-attention module (MLSA) and a global masked self-attention module (MGSA). The designs of both adopt the mechanism of the masked multi-head self-attention module in the Transformer decoder (Mask Multi-Head Attention), but the local masked attention module is slightly different. It calculates in a local window manner. Therefore, only the corresponding mask of the local window is taken during calculation, and then local window calculation (such as the local window calculation method in LSA) is performed. Therefore, a decoding module consists of: MLSA, MGSA, LSA, GSA, layer normalization, and a fully connected feed-forward network. The specific calculation process of the decoding module can be expressed by the following mathematical method:

[0092] Assume the following input is given:

[0093]

[0094] Key′ = Encoder(F′ seq )

[0095] Value′ = Encoder(F′ seq )

[0096] Then the decoding module performs the following calculations:

[0097] D mloc = MLSA(Query′ i , Key′, Value′|winsize, mask)

[0098] D mglobal = MGSA(Query′ i , Key′, Value′|mask)

[0099] D loc = LSA(LN(D mloe + Query′ i ), Key′, Value′|winsize)

[0100] D global = GSA(LN(D mglobal + Query′ i ), Key′, Value′)

[0101] D fusion = Concat(D loc , D global ) c ·W′0

[0102] Query′ i+1 = LN(Query′ i + FFN(LN(Query′ i + D fusion )))

[0103] where, Embedding( <start>) is the start symbol of the encoded video. The video start symbol is defined as the starting frame of a blank image (all pixel values are 0), mask is the mask image, and W′0 is a learnable parameter matrix.

[0104] It should be noted here that after the encoder-decoder module processes the result, it inputs the result into the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature, obtaining multiple ordered combined probabilities; then calls the behavior classifier to identify the multiple ordered combined probabilities to obtain the behavior detection result. However, as a transformable implementation method, the encoder-decoder can also directly perform behavior detection and recognition after processing the result. That is, a behavior classifier and a boundary regressor are set after the encoder-decoder module, as Figure 2 shown. The behavior classifier is preferably an FNN, that is, a behavior classifier based on a fuzzy neural network, to achieve behavior recognition and localization. Of course, the encoder-decoder module needs to be trained to reach the preset number of iterations to obtain an optimized behavior recognition result before being officially used. In the training stage, the loss function is calculated based on the above obtained results and combined with the true labels, and then the gradient is backpropagated for training. Since the process of applying the loss function has no obvious difference from the mainstream algorithms, it will not be introduced in detail here. In the inference stage, the result can be directly output.

[0105] Of course, in order to obtain a more refined behavior detection result, after the encoder-decoder module processes the result, it can further call the embedded boundary matching nomination network module, as Figure 3 shown. The embedded boundary matching nomination network module performs boundary matching nomination processing on it to obtain multiple ordered combined probabilities. Then call the cascaded boundary refinement module to perform refinement operations on the multiple ordered combined probabilities to obtain a refined boundary result. Finally, call the behavior classifier to identify the refined boundary result to obtain the behavior detection result. As Figure 3 shown, the features extracted from the backbone network and spliced can be simultaneously fed into the encoder-decoder module and the embedded boundary matching nomination network module, and finally the processing results of both are simultaneously fed into the behavior classifier to obtain an effective behavior detection result. The behavior classifier includes a behavior classifier based on a fuzzy neural network and a behavior classifier applicable to nomination segments.

[0106] Embodiment 3:

[0107] This embodiment implements a behavior detection method based on an embedded arbitrary boundary matching nomination network, and uses a behavior detection device based on the embedded arbitrary boundary matching nomination network for behavior detection. Among them, the behavior detection device based on the embedded arbitrary boundary matching nomination network includes a backbone network, an encoder-decoder module, an embedded boundary matching nomination network module, and a behavior classifier; the method includes: the first step, calling the backbone network to extract features from the unclipped video to obtain the first feature; the second step, calling the encoder-decoder module to encode and decode the first feature to obtain the second feature; the third step, calling the embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain multiple ordered combined probabilities; the fourth step, calling the cascaded boundary refinement module to refine the multiple ordered combined probabilities to obtain a fine boundary result; the fifth step, calling the behavior classifier to identify the fine boundary result to obtain a behavior detection result.

[0108] In a possible specific implementation, assume that F' seq is fed into the trained embedded boundary matching nomination network module and there are three outputs: They are the segment start point prediction probability, the segment end point prediction probability, and the boundary matching confidence map respectively. After obtaining the three outputs, it is necessary to traverse each start point and calculate its combined probability p with all end points f that meet the requirements and are less than the actual time series length l i，j . Then, combined with the boundary matching confidence map (l d is the maximum time series length), multiply it by the confidence score s i，j-1 at the corresponding position to obtain the confidence g i，j of the "i-th start point - j-th end point" combination. Finally, a total of m = l f 2 -(1 + l f ) * l f * 0.5 combinations and combination scores G ∈ R m are obtained. After obtaining the scores of all combinations, it is necessary to sort all the scores in descending order, then select the top n "start - end" combinations as the selected candidate nomination targets, then use RoI Align to sample the corresponding local features according to the candidate nomination targets, and through the above matching mechanism, pair the candidate targets and the real annotation information one by one, and finally send them into the behavior classifier and the boundary refinement module together. The calculation steps are as follows:

[0109]

[0110] where \(i\in[0,\ldots,l f -1]\), \(j > i\) and \(j < l f

[0111] g i，j = p i，j * s i，j-1

[0112] \(G(\text{start},\text{end})=\{g 0，1 , g 0，2 , g 0，3 \ldots, g 0，99 , g 12 , \ldots, g 98，99 \}\

[0113] Proposals = \text{SelectTopN}(G(\text{start},\text{end}))

[0114] Since most mainstream nomination algorithms are not implemented considering only the engineering efficiency level, most of the above post - processing methods are implemented in the form of for - loops. The time complexity of the for - loop method is \(O(l f 2 )\). It can be seen that the time complexity of this method increases with the increase of the time series length, which is not conducive to embedding the nomination network into the detection framework. Therefore, this invention uses parallel computing to implement this post - processing operation. Specifically, multiply \(P start \) and \((P end ) τ \) element - by - element to obtain the "start - end" combined confidence map To meet the and less than \(l f \) constraint conditions, perform the Hadamard product operation on \(P(\text{start},\text{end})\) with the all - 1 upper triangular matrix and the all - 1 matrix with a diagonal of 0 to obtain \(P(\text{start},\text{end})\) that is upper triangular and has a diagonal of 0 * , that is, all numbers in the upper triangle are valid values, while all numbers in the lower triangle and on the diagonal are invalid values. Then calculate with the boundary matching confidence map \(S\). From the above formula, the confidence score corresponding to the "the \(i\) - th start point - the \(j\) - th end point" combination is \(s i，j-1 \). Therefore, it is necessary to shift the boundary matching confidence map \(S\) one column to the right and then fill 0 in the 0 - th column (i.e., the first column) to obtain At the same time, it is also necessary to fill 0 after the last column of \(P(\text{start},\text{end}) * \) to obtain Finally, \(S * \) and \(P(\text{start},\text{end}) * Perform the Hadamard product operation to obtain the final confidence score map G(start, end). Finally, perform a descending row sorting on the confidence map G, and then select the top n / l f "start - end" combinations in each row to form the selected candidate targets. According to the n candidate targets, use RoI Align (Region of Interest Alignment) to sample the corresponding local features, and through the above matching mechanism, pair the candidate targets and the true annotation information one by one, and finally send them into the cascaded boundary refinement module together. The parallelized calculation formula is as follows:

[0115] P(start, end) = P start ·(P end ) T

[0116]

[0117] G(start, end) = P(start, end) ** ☉S *

[0118] Proposals = SelectTopN(G(start, end))

[0119] where ⊙ represents the dot product. In view of the situation of more boundary prediction noises, the present invention designs a "three - region" boundary refinement module, and through the cascaded method, realizes the refinement operation of the boundary from coarse to fine, and finally obtains a more refined segment boundary result. The following explains the single - boundary refinement module and the cascade: First, obtain the candidate nomination targets through the third point, and then perform temporal RoI Align according to the candidate nomination targets. Temporal RoI Align expands the starting local region RoI and the ending local region RoI centered on the starting boundary point and the ending boundary point respectively, and then the area between the two points is the main local region RoI; use temporal RoI Align for the three local regions and the corresponding temporal features to obtain the corresponding aligned local features F s , F c , F e . Second, input the starting local region feature and the ending local region feature into two regression models (starting regression and end regression) respectively to realize the boundary optimization of the action temporal nomination (that is, predict the offsets of the starting point and the ending point). It can be expressed by the following formula: Finally, concatenate the starting local region feature, the ending local region feature and the main local region feature to obtain F a = Concat(F s , F c , F e ) are respectively input into a regression model and a classification model. The former mainly predicts the offset of the center point of the segment, the offset of the segment length, and the probability of predicting the presence of an action, while the latter mainly predicts the corresponding action category that appears in the nomination. It can be expressed by the following formula:

[0120]

[0121] where is the probability of the behavior category, and k is the number of behavior categories. ReLU and Softmax are activation functions. Regarding the implementation of the cascade type: In this application, a total of three boundary refinement modules are used and connected in series, that is, the boundary result of the previous boundary refinement module will be used as the input of the next boundary refinement module, so as to adjust the boundary value "from rough to fine".

[0122] Embodiment 4:

[0123] This embodiment implements a behavior detection device based on an embedded arbitrary boundary matching nomination network, as Figure 4 shown. The device includes:

[0124] Backbone network 401, which is used to extract features from the unclipped video to obtain the first feature;

[0125] Encoder-decoder module 402, which is used to encode and decode the first feature to obtain the second feature;

[0126] Embedded boundary matching nomination network module 403, which is used to perform boundary matching nomination processing on the second feature to obtain multiple ordered combined probabilities;

[0127] Cascade type boundary refinement module 404, which is used to perform refinement operations on the multiple ordered combined probabilities to obtain a fine boundary result;

[0128] Behavior classifier 405, which is used to identify the fine boundary result to obtain a behavior detection result.

[0129] Preferably, the behavior classifier includes a behavior classifier based on a fuzzy neural network and a behavior classifier applicable to nomination segments.

[0130] Next, please refer to Figure 5 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As Figure 5 As shown, the electronic device 2 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected through the bus 202. A computer program that can run on the processor 200 is stored in the memory 201. When the processor 200 runs the computer program, it executes the behavior detection method provided by any of the foregoing embodiments of the present application based on an embedded arbitrary boundary matching nomination network. The electronic device may be an electronic device with a touch-sensitive display.

[0131] Among them, the memory 201 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 203 (which may be wired or wireless), a communication connection between this system network element and at least one other network element is achieved. The Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0132] The bus 202 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 201 is used to store a program. After receiving an execution instruction, the processor 200 executes the program. The behavior detection method based on an embedded arbitrary boundary matching nomination network disclosed in any of the foregoing embodiments of the present application can be applied to the processor 200 or implemented by the processor 200.

[0133] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 200 or the instructions in the form of software. The above-mentioned processor 200 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by the hardware decoding processor, or executed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and combines its hardware to complete the steps of the above method.

[0134] The electronic device provided by the embodiments of the present application and the behavior detection method based on the embedded arbitrary boundary matching nomination network provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0135] The embodiments of the present application also provide a computer-readable storage medium corresponding to the behavior detection method based on the embedded arbitrary boundary matching nomination network provided in the foregoing embodiments. Please refer to Figure 6 , Figure 6 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the behavior detection method based on the embedded arbitrary boundary matching nomination network provided in any of the foregoing embodiments.

[0136] In addition, examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here one by one.

[0137] The computer-readable storage medium provided by the above embodiments of the present application and the method for allocating quantum key distribution channels in the space division multiplexing optical network provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0138] The embodiments of the present application also provide a computer program product, including a computer program, which when executed by a processor, implements the steps of the behavior detection method based on the embedded arbitrary boundary matching nomination network provided by any of the foregoing embodiments. The steps of the method include: calling a backbone network to extract features from an unclipped video to obtain a first feature; calling an encoder-decoder module to encode and decode the first feature to obtain a second feature; calling an embedded boundary matching nomination network module to perform boundary matching nomination processing on the second feature to obtain a plurality of ordered combination probabilities; calling a behavior classifier to identify the plurality of ordered combination probabilities to obtain a behavior detection result.

[0139] It should be noted that: The algorithms and displays provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general-purpose devices can also be used in conjunction with the teachings provided herein. Based on the above description, the structures required to construct such devices are obvious. In addition, the present application is not directed to any specific programming language. It should be understood that the content of the present application described herein can be implemented using various programming languages, and the descriptions of specific languages above are for disclosing the best implementation modes of the present application. In the specification provided herein, a large number of specific details are set forth. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and technologies are not shown in detail so as not to obscure the understanding of this specification.

[0140] Similarly, it should be understood that, in order to streamline the present application and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure or description thereof. However, the disclosed method should not be construed as reflecting the intention that the claimed present application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all the features of the single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate embodiment of the present application.

[0141] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be adopted to combine all the features disclosed in this specification and all the processes or units of any method or device thus disclosed. Unless otherwise clearly stated, each feature disclosed in this specification can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0142] Each component embodiment of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation device according to the embodiments of the present application. The present application can also be implemented as a device or device program for executing part or all of the methods described herein. The program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0143] As described above, only the preferred specific embodiments of the present application are provided, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the technical field of the present application within the technical scope disclosed by the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / start>

Claims

1. A behavior detection method based on embedding arbitrary boundary matching nomination network, characterized in that Behavior detection is performed using a behavior detection device based on an embedded arbitrary boundary matching nomination network, wherein the behavior detection device based on an embedded arbitrary boundary matching nomination network includes a backbone network, an encoder-decoder module, an embedded boundary matching nomination network module, and a behavior classifier; the behavior classifier includes a behavior classifier based on a fuzzy neural network and a behavior classifier suitable for a nominated segment; the method includes: Call the backbone network to extract features from the unedited video and obtain the first feature; Sending the first feature to the encoder-decoder module and the embedded boundary matching nomination network module simultaneously, and obtaining processing results of the encoder-decoder module and the embedded boundary matching nomination network module; The processing result of the encoder-decoder module is input into the behavior classifier based on fuzzy neural network, and the processing result of the embedded boundary matching nomination network module is input into the behavior classifier applicable to the nominated segment to obtain effective behavior detection results.

2. The behavior detection method based on embedding arbitrary boundary matching nomination network according to claim 1 is characterized in that The encoder-decoder module is composed of an encoding module and a decoding module; after the backbone network is called to extract features from the unedited video and obtain the first feature, it also includes: Converting the first feature into a time series feature with time series position coding information by the encoding module; The encoder builds interdependencies between features at different temporal positions to obtain temporal high-level semantic features; The decoder extracts the corresponding temporal position information and behavior information from the temporal high-level semantic features as the second feature.

3. The behavior detection method based on embedding arbitrary boundary matching nomination network according to claim 2 is characterized in that Both the encoding module and the decoding module include a local temporal self-attention module and a global temporal self-attention module, and the decoding module also includes a local mask self-attention module and a global mask self-attention module.

4. The behavior detection method based on embedding arbitrary boundary matching nomination network according to claim 3 is characterized in that The behavior detection device based on embedding an arbitrary boundary matching nomination network further includes a cascade boundary refinement module, which is used to perform a refinement operation on multiple ordered combination probabilities corresponding to the processing results of the embedded boundary matching nomination network module.

5. The behavior detection method based on embedding arbitrary boundary matching nomination network according to claim 4 is characterized in that The refining operation on the multiple ordered combination probabilities corresponding to the processing results of the embedded boundary matching nomination network module includes: Obtain the start point prediction probability matrix and the segment end point prediction probability matrix; Perform a dot product of the start point prediction probability matrix and the transposed matrix of the segment end point prediction probability matrix to obtain a first combined confidence map; Performing a Hadamard product operation on the first combined confidence map, an upper triangular matrix with all 1s, and a matrix with all 1s and a diagonal of 0s to obtain a second combined confidence map; Based on the calculation and zero-padding operation of the first combined confidence map, the second combined confidence map, and the boundary matching confidence map, a third combined confidence map is obtained and used as a refined boundary result.

6. A behavior detection device based on an embedded arbitrary boundary matching nomination network, characterized in that: The device comprises: The backbone network is used to extract features from the unedited video to obtain the first feature; an encoder-decoder module, configured to encode and decode the first feature to obtain a processing result of the encoder-decoder module; an embedded boundary matching nomination network module, configured to perform boundary matching nomination processing on the first feature to obtain a processing result of the embedded boundary matching nomination network module; A cascade boundary refinement module is used to refine a plurality of ordered combination probabilities corresponding to the processing results of the embedded boundary matching nomination network module to obtain a refined boundary result; A behavior classifier is used to identify the fine boundary result and obtain a behavior detection result; the behavior classifier includes a behavior classifier based on a fuzzy neural network and a behavior classifier suitable for the nominated segment; The device is used to implement the following steps: Sending the first feature to the encoder-decoder module and the embedded boundary matching nomination network module simultaneously, and obtaining processing results of the encoder-decoder module and the embedded boundary matching nomination network module; The processing result of the encoder-decoder module is input into the behavior classifier based on fuzzy neural network, and the processing result of the embedded boundary matching nomination network module is input into the behavior classifier applicable to the nominated segment to obtain effective behavior detection results.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images