Identification method, system and electronic device based on trust risk allocation token
Through the identification method of trust risk allocation tokens, Vision Transformer and 3D convolution are used to extract spatiotemporal features, combined with fast Fourier transform and evidence theory, distinguish between active and passive motion in drone video, and solve the recognition performance when camera moves The problem of declining improves the accuracy and efficiency of behavior recognition.
Patent Information
- Application Number
- CN202411059390.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-03
Smart Images

Figure CN118570711B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video recognition technology, and in particular to an identification method, system and electronic equipment based on trust risk allocation tokens. Background Art
[0002] With the rapid development of social security network video surveillance systems, the emergence of large-scale video surveillance data has become a challenge. At the same time, shooting under high-altitude and high-point cameras brings many conveniences to the study of crowd gathering behavior and traffic flow analysis under the overall situation. It is crucial to use the perspective of drones to study efficient video image analysis technology to extract abnormal behavior information from large-scale video surveillance data. The development of this technology can not only effectively cope with the challenges brought by the growth of video data and save the cost of public security prevention and control, but also achieve a wider monitoring range and more efficient abnormal behavior detection through behavior recognition from the perspective of drones. Behavior recognition from the perspective of drones makes full use of the advantages of drones to achieve remote observation and recognition of target behaviors, which not only improves the efficiency and accuracy of observation, but also reduces the demand and risk of human resources. By applying behavior recognition technology from the perspective of drones to various fields, more support and guarantee can be provided for the development and security of human society.
[0003] However, in the existing drone behavior recognition technology, when the camera moves, a large amount of passive motion information appears in the video due to the movement and rotation of the camera. It is difficult for the model to distinguish whether the motion information comes from the subject's action or the result of camera rotation, which causes significant interference to behavior recognition. Especially in drone videos where the proportion of people is small and the proportion of background is extremely large, all motion information is treated equally, and it is impossible to effectively deal with the interference of passive motion information, which affects the recognition performance.
[0004] Therefore, finding a method that can quantify the uncertainty of passive motion information and improve the accuracy of behavior recognition is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention
[0005] In view of this, the present invention proposes an identification method, system and electronic device based on trust risk allocation tokens, which can use fast Fourier transform to perform frequency domain analysis on intermediate features, effectively distinguish active tokens and passive tokens in video sequences, make full use of the complementary information of the two types of tokens, obtain comprehensive behavior representation, and improve recognition accuracy and efficiency.
[0006] The technical solution of the present invention is achieved in this way:
[0007] In a first aspect, the present invention provides an identification method based on trust risk allocation token, comprising the following steps:
[0008] S1. Obtain a video sequence from the drone's perspective, input the video sequence into the Vision Transformer backbone network, use 3D convolution to extract spatiotemporal features of the video sequence to obtain a spatiotemporal representation, and perform feature extraction and category score prediction on the spatiotemporal features to obtain intermediate features and category scores of the intermediate features;
[0009] S2. Use fast Fourier transform to analyze the frequency domain information of the intermediate features to obtain the dynamic significance level score;
[0010] S3, dividing the tokens in the intermediate features into active tokens and passive tokens according to the dynamic significance level scores;
[0011] S4, using evidence theory and Dirichlet distribution, calculate the prediction uncertainty score of each passive token through the Gumbel-Softmax function;
[0012] S5. Sort and filter the passive tokens according to the prediction uncertainty scores of the passive tokens, optimize the passive tokens, and obtain optimized passive tokens;
[0013] S6. Use the Vision Transformer backbone network to extract features from active tokens and optimized passive tokens to obtain the behaviors in the video sequence.
[0014] Based on the above technical solutions, preferably, the Vision Transformer backbone network includes 12 layers of Transformer blocks, and the Transformer blocks include local multi-head self-attention, global multi-head self-attention, a feedforward network and a fully connected layer.
[0015] On the basis of the above technical solution, preferably, step S1 specifically includes:
[0016] S11. Obtain the video sequence from the drone’s perspective, use 3D convolution and Vision Transformer backbone network to extract the spatiotemporal features of the video sequence, and obtain the spatiotemporal representation;
[0017] S12, use the multi-head self-attention and feed-forward network of the first 6 layers of Transformer blocks to extract spatiotemporal features and obtain intermediate features;
[0018] S13. Use a multi-layer perceptron to process the intermediate features, predict the category to which each spatiotemporal representation belongs, and obtain the category score of the intermediate features.
[0019] On the basis of the above technical solution, preferably, step S2 specifically includes:
[0020] S21, performing frequency sampling on the intermediate features in the time dimension to obtain a frequency representation, and converting the time domain signal into a frequency domain signal;
[0021] S22. Calculate the amplitude of the time domain signal at each space and channel position at different frequencies using fast Fourier transform. The calculation formula is as follows:
[0022]
[0023] in, represents the amplitude of the time domain signal at different frequencies, represents the intermediate feature, Represents the sampling point index of the time domain signal, Represents the total number of sampling points of the time domain signal, Represents the index in the frequency domain, corresponding to different frequency components;
[0024] S23, calculating the dynamic significance level score of each token according to the amplitude and frequency of the time domain signal at each spatial and channel position at different frequencies, wherein the intermediate feature includes at least one token, and the calculation formula of the dynamic significance level score is as follows:
[0025]
[0026] in, represents the dynamic significance level score of the token, Indicates the frequency of sampling of intermediate features in the time dimension, represents the L2 norm.
[0027] More preferably, step S5 specifically includes:
[0028] S51, sorting the passive tokens from low to high according to the prediction uncertainty scores of the passive tokens to form a passive token sequence;
[0029] S52, setting a first uncertainty score threshold and a second uncertainty score threshold, wherein the second uncertainty score threshold is greater than the first uncertainty score threshold;
[0030] S53, screening the passive token sequence according to the first uncertainty score threshold and the second uncertainty score threshold to obtain an optimized passive token sequence:
[0031] Retaining passive tokens whose uncertainty scores are less than a first uncertainty score threshold;
[0032] semantically merging passive tokens whose uncertainty scores are greater than a first uncertainty score threshold but less than a second uncertainty score threshold;
[0033] Deleting passive tokens whose uncertainty scores are greater than a second uncertainty score threshold;
[0034] The passive tokens whose uncertainty scores are less than the first uncertainty score threshold and the semantically merged passive tokens form an optimized passive token sequence.
[0035] On the basis of the above technical solution, preferably, the semantic merging specifically includes:
[0036] The density peak clustering and nearest neighbor search method are used to determine the category of each passive token;
[0037] Passive tokens with the same category are weightedly merged to obtain semantically merged passive tokens.
[0038] Based on the above technical solution, preferably, the method of using density peak clustering and nearest neighbor search to determine the fine-grained category of each passive token specifically includes:
[0039] Select the K nearest neighbors of each passive token according to the Mahalanobis distance and calculate the local density of each token using the Gaussian kernel function and the Mahalanobis distance metric;
[0040] Calculate the distance between each passive token and the passive token corresponding to a higher local density respectively, and form a distance matrix;
[0041] confirming the token distance indicator based on the distance matrix;
[0042] The passive token with the highest local density is set as the maximum distance between the current passive token and other passive tokens, the local density of the current passive token and the token distance indicator are combined, the score of each passive token is evaluated using a score function, and the passive token with the highest score is taken as the cluster representative;
[0043] Assign other passive tokens to the nearest cluster representative to obtain the category and cluster results for each token.
[0044] On the basis of the above technical solution, preferably, the calculation formula of the token distance indicator is as follows:
[0045]
[0046] in, represents the distance indicator of token i, represents the local density of any token i, represents the local density of any token j, Representation Token The corresponding token features, Representation Token The corresponding token features, Represents the covariance matrix of a multidimensional random variable;
[0047] The local density is calculated as follows:
[0048]
[0049] in, Representation Token of A close neighbor.
[0050] In a second aspect, the present invention provides an identification system based on trust risk allocation tokens, using the identification method as described above, comprising:
[0051] A first feature extraction module is configured to obtain a video sequence from the perspective of the drone, input the video sequence into the Vision Transformer backbone network, and use 3D convolution to extract spatiotemporal features of the video sequence to obtain a spatiotemporal representation;
[0052] A second feature extraction module is configured to perform feature extraction and category score prediction on the spatiotemporal features to obtain intermediate features and category scores of the intermediate features;
[0053] A frequency domain analysis module configured to use a fast Fourier transform to perform frequency domain information analysis on the intermediate features to obtain a dynamic significance level score;
[0054] a token classification module configured to classify tokens in the intermediate features into active tokens and passive tokens based on a dynamic significance level score;
[0055] an uncertainty calculation module configured to calculate a prediction uncertainty score for each passive token through a Gumbel-Softmax function using evidence theory and Dirichlet distribution;
[0056] a token optimization module configured to sort and filter the passive tokens according to the predicted uncertainty scores of the passive tokens, optimize the passive tokens, and obtain optimized passive tokens;
[0057] The behavior recognition module is configured to use the Vision Transformer backbone network to extract features from active tokens and optimized passive tokens to obtain behaviors in video sequences.
[0058] In a third aspect, the present invention provides an electronic device, comprising: at least one processor, at least one memory, a communication interface, and a bus;
[0059] Wherein, the processor, memory, and communication interface communicate with each other via the bus;
[0060] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the identification method as described above.
[0061] The identification method of the present invention has the following beneficial effects compared with the prior art:
[0062] By using the Vision Transformer backbone network and 3D convolution to extract spatiotemporal features from video sequences, we obtain spatiotemporal representations and intermediate features. We use fast Fourier transform to perform frequency domain analysis on intermediate features, effectively distinguishing active tokens from passive tokens in video sequences, making full use of the complementary information of the two types of tokens, and obtaining a comprehensive behavior representation, thus improving recognition accuracy and efficiency.
[0063] By sorting and screening passive tokens, the passive token sequence is optimized, the accuracy and efficiency of behavior recognition are improved, and the first uncertainty score threshold and the second uncertainty score threshold are set to achieve a balance between retaining key information and filtering noise, thereby improving the performance and efficiency of the entire behavior recognition method;
[0064] By calculating the Mahalanobis distance between each passive token and its K nearest neighbors and using the Gaussian kernel function for local density estimation, the similarities and differences between tokens are captured, and the token distance indicator is used to quantify the degree of association between each passive token and other passive tokens, achieving adaptive classification of tokens and thus improving the accuracy of behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0066] Figure 1 is a flow chart of the identification method of the present invention;
[0067] Figure 2 It is a block diagram of the identification method of the present invention. DETAILED DESCRIPTION
[0068] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0069] like Figure 1 and Figure 2 As shown, the present invention provides an identification method based on trust risk allocation token, comprising the following steps:
[0070] S1. Obtain the video sequence from the drone's perspective, input the video sequence into the Vision Transformer backbone network, use 3D convolution to extract spatiotemporal features of the video sequence, obtain the spatiotemporal representation Xin, and perform feature extraction and category score prediction on the spatiotemporal features to obtain the intermediate features and intermediate features The category score of .
[0071] Furthermore, the Vision Transformer backbone network includes 12 layers of Transformer blocks, and the Transformer blocks include local multi-head self-attention, global multi-head self-attention, a feedforward network and a fully connected layer.
[0072] It can be understood that the local multi-head self-attention mechanism can capture the spatiotemporal dependencies within local areas in the video sequence and extract local features; the global multi-head self-attention mechanism can model the long-range dependencies between different positions in the entire video sequence and extract global features; the feedforward network and the fully connected layer are used to perform feature transformation and information fusion to improve the nonlinear ability of feature representation.
[0073] Specifically, step S1 specifically includes:
[0074] S11, obtain the video sequence from the drone’s perspective, use 3D convolution and Vision Transformer backbone network to extract the spatiotemporal features of the video sequence, and obtain the spatiotemporal representation Xin;
[0075] S12, use the multi-head self-attention and feed-forward network of the first 6 layers of Transformer blocks to extract features from the spatiotemporal representation Xin and obtain intermediate features ;
[0076] S13. Use multi-layer perceptron to analyze intermediate features Process and predict each space-time X in Indicates the category to which it belongs and obtains the category score of the intermediate feature.
[0077] As you can understand, a multilayer perceptron (MLP) is a feedforward neural network consisting of multiple linear layers and nonlinear activation functions. Mapped to the category space and obtained the category score of the intermediate feature.
[0078] By using 3D convolution and Vision Transformer backbone networks to extract spatiotemporal features of video sequences, the temporal and spatial information in the video can be effectively captured. The multi-head self-attention mechanism captures the relationship and dependency between spatiotemporal features from different subspaces and angles, and extracts diversified feature representations. The feedforward network performs nonlinear transformation and information fusion on the extracted features to enhance the representation capability of the features, which helps to improve the accuracy and robustness of behavior recognition.
[0079] S2. Use fast Fourier transform to transform intermediate features Perform frequency domain information analysis to obtain dynamic significance level scores.
[0080] Specifically, step S2 specifically includes:
[0081] S21. Intermediate features Perform frequency sampling in the time dimension to obtain the frequency representation freq, and convert the time domain signal into the frequency domain signal; where freq∈R c , R c Indicates the number of channels for frequency representation;
[0082] S22. Calculate the amplitude of the time domain signal at each space and channel position at different frequencies using fast Fourier transform. The calculation formula is as follows:
[0083]
[0084] in, represents the amplitude of the time domain signal at different frequencies, represents the intermediate feature, Represents the sampling point index of the time domain signal, Represents the total number of sampling points of the time domain signal, Represents the index in the frequency domain, corresponding to different frequency components;
[0085] S23, calculate the dynamic significance level score of each token according to the amplitude and frequency of the time domain signal at each spatial and channel position at different frequencies, where the intermediate feature Including at least one token, the dynamic significance level score is calculated as follows:
[0086]
[0087] in, represents the dynamic significance level score of the token, Indicates the frequency of sampling of intermediate features in the time dimension, represents the L2 norm.
[0088] Understandably, tokens represent intermediate features The elements in It can be regarded as a sequence or set, which contains multiple tokens, each of which represents the feature information of a specific spatiotemporal position in the video sequence.
[0089] By converting the time domain signal into the frequency domain signal, the frequency domain information reflects the intensity and distribution of different frequency components in the video sequence, including the periodicity and change characteristics of the motion; and the amplitude reflects the importance and contribution of the frequency component in the time domain signal. According to the amplitude and frequency of the time domain signal at different frequencies in each space and channel position, the dynamic significance level score of each token is calculated to quantify the dynamic significance of each token in the time dimension. The higher the dynamic significance score, the more significant the motion change of the token in the video sequence, and the greater the contribution of the corresponding frequency component.
[0090] In one embodiment of the present application, the fast Fourier transform uses an orthogonal normalized fast Fourier transform, which helps to extract independent frequency domain features, enhance the discriminative ability of feature representation, and improve computational efficiency, which helps to better capture dynamic information in video sequences.
[0091] S3, according to the dynamic significance level score, the intermediate features The tokens in are divided into active tokens and passive tokens.
[0092] The intermediate features are divided according to the dynamic significance level score by the set threshold μ The tokens in are divided into active tokens and passive tokens, where active tokens are tokens whose dynamic saliency level scores exceed the preset threshold μ, and active tokens correspond to areas in the video sequence with significant motion and rich information, such as moving objects, gestures, etc.; passive tokens are tokens whose dynamic saliency level scores do not reach the preset threshold μ, and passive tokens correspond to areas in the video sequence that are static or have insignificant motion, such as background, noise, etc.
[0093] In behavior recognition, active tokens carry more dynamic information related to the behavior and are therefore more important, while passive tokens may contain some redundant or irrelevant information and contribute less to the behavior recognition task.
[0094] The threshold μ is set according to actual usage requirements, and this application does not impose any specific restrictions on this.
[0095] S4. Using evidence theory and Dirichlet distribution, the prediction uncertainty score of each passive token is calculated through the Gumbel-Softmax function.
[0096] Specifically, we use the Dirichlet distribution method combined with the evidence theory (DST) , apply the Gumbel-Softmax function to process the category scores of the intermediate features and obtain evidence e k ;
[0097] According to DST and SL theory (subjective logic theory), the parameters of Dirichlet distribution are With evidence k ,by The equation of is connected to form an evidence-based class probability model, and the calculation formula of the prediction uncertainty score is as follows:
[0098]
[0099] in, represents the uncertainty score, k represents the number of categories, and S represents The sum.
[0100] As can be understood, evidence theory (DST) is a mathematical framework for dealing with uncertainty and incomplete information, which provides information about the category to which the passive token belongs; Dirichlet distribution is a multivariate probability distribution used to model the category probability distribution of passive tokens; Gumbel-Softmax function is used to convert the category score of the intermediate feature into evidence ek.
[0101] By calculating the uncertainty score of each passive token, the uncertainty score is used to quantify the degree of uncertainty of the passive token in category prediction, that is, the higher the score, the greater the uncertainty of the prediction and the more ambiguous the category attribution of the passive token; by linking the parameters of the Dirichlet distribution with the evidence, the probability distribution of the passive token can be better captured, thereby improving the accuracy and reliability of behavior recognition.
[0102] S5. Sort and screen the passive tokens according to their predicted uncertainty scores, optimize the passive tokens, and obtain optimized passive tokens.
[0103] Specifically, step S5 specifically includes:
[0104] S51, sorting the passive tokens from low to high according to the prediction uncertainty scores of the passive tokens to form a passive token sequence;
[0105] S52, setting a first uncertainty score threshold and a second uncertainty score threshold, wherein the second uncertainty score threshold is greater than the first uncertainty score threshold;
[0106] S53, screening the passive token sequence according to the first uncertainty score threshold and the second uncertainty score threshold to obtain an optimized passive token sequence:
[0107] Retaining passive tokens whose uncertainty scores are less than a first uncertainty score threshold;
[0108] semantically merging passive tokens whose uncertainty scores are greater than a first uncertainty score threshold but less than a second uncertainty score threshold;
[0109] Deleting passive tokens whose uncertainty scores are greater than a second uncertainty score threshold;
[0110] The passive tokens whose uncertainty scores are less than the first uncertainty score threshold and the semantically merged passive tokens form an optimized passive token sequence.
[0111] It can be understood that the first uncertainty score threshold and the second uncertainty score threshold are set according to actual usage requirements, and the present application does not make any specific limitations on this.
[0112] By sorting and filtering passive tokens, the passive token sequence is optimized, and the accuracy and efficiency of behavior recognition are improved. By setting the first uncertainty score threshold and the second uncertainty score threshold, a balance is achieved between retaining key information and filtering noise, which can better adapt to video sequences of different quality and complexity. The optimized passive token sequence provides a more refined and information-rich input for subsequent behavior recognition, thereby improving the performance and efficiency of the entire behavior recognition method.
[0113] Specifically, step S5 is described with a specific embodiment:
[0114] There are 10 passive tokens (token A: 0.1, token B: 0.2, token C: 0.3, token D: 0.4, token E: 0.5, token F: 0.6, token G: 0.7, token H: 0.8, token I: 0.9 and token J: 1.0), set the first uncertainty score threshold to 0.3 and the second uncertainty score threshold to 0.7;
[0115] Sort the passive tokens from low to high according to the prediction uncertainty score to form a token sequence: Token A (0.1) < Token B (0.2) < Token C (0.3) < Token D (0.4) < Token E (0.5) < Token F (0.6) < Token G (0.7) < Token H (0.8) < Token I (0.9) < Token J (1.0);
[0116] Filter token sequences based on a threshold:
[0117] Tokens A, B, and C whose uncertainty scores are less than the first threshold (0.3) are retained;
[0118] Semantically merge tokens D, E, and F whose uncertainty scores are greater than the first threshold (0.3) but less than the second threshold (0.7) to obtain a new token K;
[0119] Delete tokens G, H, I, and J whose uncertainty scores are greater than the second threshold (0.7);
[0120] The optimized passive token sequence is: token A, token B, token C, token K.
[0121] Furthermore, the semantic merging specifically includes:
[0122] The density peak clustering and nearest neighbor search method are used to determine the category of each passive token;
[0123] Passive tokens with the same category are weightedly merged to obtain semantically merged passive tokens.
[0124] Among them, the calculation formula for weighted merging of attention is as follows:
[0125]
[0126] represents the feature representation of the merged passive tokens, represents the set of the e-th cluster in the density peak clustering, Represents intermediate features The feature vector of each token in Represents weight.
[0127] It can be understood that tokens represent local features further extracted on the spatiotemporal representation, where the clustered passive tokens are weighted using a linear layer, i.e. Represents the weights of the linear layer.
[0128] Through density peak clustering and nearest neighbor search, semantically similar or related passive tokens are clustered together to form more compact and cohesive categories, so as to better capture the semantic relationship between passive tokens and improve the accuracy of semantic representation; through attention-weighted merging of passive tokens of the same category, redundant information is removed and the robustness of semantic representation is enhanced. Even if the semantic representation of a passive token is noisy or inaccurate, the impact can be mitigated by merging it with other similar tokens, thereby improving the overall quality of semantic representation.
[0129] Furthermore, the method of using density peak clustering and nearest neighbor search to determine the category of each passive token specifically includes:
[0130] Select the K nearest neighbors of each passive token according to the Mahalanobis distance and calculate the local density of each token using the Gaussian kernel function and the Mahalanobis distance metric;
[0131] Calculate the distance between each passive token and the passive token corresponding to a higher local density respectively, and form a distance matrix;
[0132] Confirm the token distance indicator based on the distance matrix ;
[0133] The passive token with the highest local density is set as the maximum distance between the current passive token and other passive tokens, the local density of the current passive token and the token distance indicator are combined, the score of each passive token is evaluated using a score function, and the passive token with the highest score is taken as the cluster representative;
[0134] Assign other passive tokens to the nearest cluster representative to obtain the category and cluster results for each token.
[0135] It can be understood that each row in the distance matrix corresponds to a passive token, each column corresponds to another passive token with a local density higher than the current passive token, and the distance matrix element is the Mahalanobis distance between the two passive tokens.
[0136] By calculating the Mahalanobis distance between each passive token and its K nearest neighbors and using the Gaussian kernel function for local density estimation, the distribution of passive tokens in the feature space can be accurately reflected, and the similarities and differences between tokens can be captured; the token distance indicator is used to quantify the degree of association between each passive token and other passive tokens, and the maximum distance and token distance indicator are evaluated using a scoring function to determine the cluster center, and other passive tokens are assigned to the nearest cluster representative, realizing adaptive classification of tokens, obtaining optimized clustering results, and realizing fine-grained semantic clustering of passive tokens, thereby improving the accuracy of behavior recognition.
[0137] Specifically, the calculation formula of the token distance indicator is as follows:
[0138]
[0139] in, represents the distance indicator of token i, represents the local density of any token i, represents the local density of any token j, Representation Token The corresponding token features, Representation Token The corresponding token features, Represents the covariance matrix of a multidimensional random variable;
[0140] The local density is calculated as follows:
[0141]
[0142] in, Representation Token of A close neighbor.
[0143] In an embodiment of the present application, the passive token having an uncertainty score greater than a second uncertainty score threshold is deleted, further comprising:
[0144] Define the deletion token distance indicator D(i), where
[0145]
[0146] Indicates that all uncertainty scores are less than the token gather, Represents the element number threshold, if The number of elements in is less than ,but Indicates the deletion token ;if The number of elements in is greater than or equal to ,but Retains the token .
[0147] The element quantity threshold is used to adaptively decide whether to delete passive tokens, control the intensity of deletion, balance information utilization and noise suppression, retain valuable information, and improve the reliability of optimization.
[0148] A specific embodiment is used to illustrate:
[0149] Passive tokens with uncertainty scores greater than a second uncertainty score threshold include token 7 (uncertainty score 0.7), token 8 (uncertainty score 0.8), token (uncertainty score 0.9), and token 10 (uncertainty score 1.0);
[0150] For token 7, S7 = {token 1, token 2, token 3, token 4, token 5, token 6};
[0151] For token 8, S8 = {token 1, token 2, token 3, token 4, token 5, token 6, token 7};
[0152] For token 9, S9 = {token 1, token 2, token 3, token 4, token 5, token 6, token 7, token 8};
[0153] For token 10, S10 = {token 1, token 2, token 3, token 4, token 5, token 6, token 7, token 8, token 9}.
[0154] When the element number threshold r is 7, S7=6<r, so delete token 7; S8=7=r, so keep token 8; S9=9>r, so keep token 9; S10=9>r, so keep token 10.
[0155] S6. Use the Vision Transformer backbone network to extract features from active tokens and optimized passive tokens to obtain the behaviors in the video sequence.
[0156] Specifically, the multi-head self-attention and feed-forward network in the last 6 layers of Transformer blocks in the Vision Transformer backbone network are used for feature extraction.
[0157] By utilizing the Vision Transformer backbone network and 3D convolution to extract spatiotemporal features from video sequences, we obtain spatiotemporal representations and intermediate features. We use fast Fourier transform to perform frequency domain analysis on the intermediate features, effectively distinguishing active tokens from passive tokens in video sequences to reduce redundancy and noise interference. We use uncertainty scores to quantitatively evaluate the reliability of each passive token for behavior recognition, optimize passive tokens, filter out redundant and noisy tokens, and finally extract features from active tokens and optimized passive tokens. We make full use of the complementary information of the two types of tokens, obtain a comprehensive behavior representation, and improve recognition accuracy and efficiency.
[0158] In one embodiment of the present application, the TOP-1 accuracy rate and the TOP-5 accuracy rate are used to evaluate the accuracy of the recognition method. The higher the accuracy rate, the better the effectiveness of the recognition method.
[0159] On the other hand, the present invention provides an identification system based on trust risk allocation tokens, using the above-mentioned identification method, including:
[0160] A first feature extraction module is configured to obtain a video sequence from the perspective of the drone, input the video sequence into the Vision Transformer backbone network, perform video processing and spatiotemporal feature extraction on the video sequence using 3D convolution, and obtain a spatiotemporal representation;
[0161] A second feature extraction module is configured to perform feature extraction and category score prediction on the spatiotemporal features to obtain intermediate features and category scores of the intermediate features;
[0162] A frequency domain analysis module configured to use a fast Fourier transform to perform frequency domain information analysis on the intermediate features to obtain a dynamic significance level score;
[0163] a token classification module configured to classify tokens in the spatiotemporal features into active tokens and passive tokens according to a dynamic significance level score;
[0164] an uncertainty calculation module configured to calculate a prediction uncertainty score for each passive token through a Gumbel-Softmax function using evidence theory and Dirichlet distribution;
[0165] a token optimization module configured to sort and filter the passive tokens according to the predicted uncertainty scores of the passive tokens, optimize the passive tokens, and obtain optimized passive tokens;
[0166] The behavior recognition module is configured to use the Vision Transformer backbone network to extract features from active tokens and optimized passive tokens to obtain behaviors in video sequences.
[0167] The present invention also provides an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the identification method described above.
[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for identifying trust risk allocation tokens, characterized in that: The following steps are involved: S1. Obtain a video sequence from the drone's perspective, input the video sequence into the Vision Transformer backbone network, use 3D convolution to extract spatiotemporal features of the video sequence to obtain a spatiotemporal representation, and perform feature extraction and category score prediction on the spatiotemporal features to obtain intermediate features and category scores of the intermediate features; S2. Use fast Fourier transform to analyze the frequency domain information of the intermediate features to obtain the dynamic significance level score; S3, dividing the tokens in the intermediate features into active tokens and passive tokens according to the dynamic significance level scores; S4, using evidence theory and Dirichlet distribution, calculate the prediction uncertainty score of each passive token through the Gumbel-Softmax function; S5. Sort and filter the passive tokens according to the prediction uncertainty scores of the passive tokens, optimize the passive tokens, and obtain optimized passive tokens; S6. Use the Vision Transformer backbone network to extract features from active tokens and optimized passive tokens to obtain behaviors in the video sequence. Step S2 specifically includes: S21. Intermediate features Perform frequency sampling in the time dimension to obtain the frequency representation freq, and convert the time domain signal into the frequency domain signal; where freq∈R c , R c Indicates the number of channels for frequency representation; S22. Calculate the amplitude of the time domain signal at each space and channel position at different frequencies using fast Fourier transform. The calculation formula is as follows: ; in, represents the amplitude of the time domain signal at different frequencies, represents the intermediate feature, Represents the sampling point index of the time domain signal, Represents the total number of sampling points of the time domain signal, Represents the index in the frequency domain, corresponding to different frequency components; S23, calculate the dynamic significance level score of each token according to the amplitude and frequency of the time domain signal at each spatial and channel position at different frequencies, where the intermediate feature Including at least one token, the dynamic significance level score is calculated as follows: ; in, represents the dynamic significance level score of the token, Indicates the frequency of sampling of intermediate features in the time dimension, represents the L2 norm; Tokens represent intermediate features Elements in, intermediate features It contains multiple tokens, each token represents the feature information of a specific spatiotemporal position in the video sequence; the active token is a token whose dynamic significance level score exceeds the preset threshold μ, and the passive token is a token whose dynamic significance level score does not reach the preset threshold μ; The evidence theory DST is combined with Dirichlet distribution, and the Gumbel-Softmax function is applied to process the category scores of the intermediate features to obtain the evidence e k ; According to DST and subjective logic SL theory, the parameters of Dirichlet distribution are With evidence k ,by The equation of is connected to form an evidence-based class probability model, and the calculation formula of the prediction uncertainty score is as follows: ; in, represents the uncertainty score, k represents the number of categories, and S represents of and; Step S5 specifically includes: S51, sorting the passive tokens from low to high according to the prediction uncertainty scores of the passive tokens to form a passive token sequence; S52, setting a first uncertainty score threshold and a second uncertainty score threshold, wherein the second uncertainty score threshold is greater than the first uncertainty score threshold; S53, screening the passive token sequence according to the first uncertainty score threshold and the second uncertainty score threshold to obtain an optimized passive token sequence: Retaining passive tokens whose uncertainty scores are less than a first uncertainty score threshold; semantically merging passive tokens whose uncertainty scores are greater than a first uncertainty score threshold but less than a second uncertainty score threshold; Deleting passive tokens whose uncertainty scores are greater than a second uncertainty score threshold; The passive tokens whose uncertainty scores are less than the first uncertainty score threshold and the semantically merged passive tokens form an optimized passive token sequence.
2. A method for identifying trust risk allocation tokens according to claim 1, characterized in that: The Vision Transformer backbone network includes 12 layers of Transformer blocks, and the Transformer blocks include local multi-head self-attention, global multi-head self-attention, feedforward network and fully connected layers.
3. A method for identifying trust risk allocation tokens according to claim 2, characterized in that: Step S1 specifically includes: S11. Obtain the video sequence from the drone’s perspective, use 3D convolution and Vision Transformer backbone network to extract the spatiotemporal features of the video sequence, and obtain the spatiotemporal representation; S12, use the multi-head self-attention and feed-forward network of the first 6 layers of Transformer blocks to extract spatiotemporal features and obtain intermediate features; S13. Use a multi-layer perceptron to process the intermediate features, predict the category to which each spatiotemporal representation belongs, and obtain the category score of the intermediate features.
4. The identification method based on trust risk allocation token according to claim 1, characterized in that: The semantic merging specifically includes: The density peak clustering and nearest neighbor search method are used to determine the category of each passive token; Passive tokens with the same category are weightedly merged to obtain semantically merged passive tokens.
5. The identification method based on trust risk allocation token according to claim 4, characterized in that: The method of using density peak clustering and nearest neighbor search to determine the fine-grained category of each passive token specifically includes: Select the K nearest neighbors of each passive token according to the Mahalanobis distance and calculate the local density of each token using the Gaussian kernel function and the Mahalanobis distance metric; Calculate the distance between each passive token and the passive token corresponding to a higher local density respectively, and form a distance matrix; confirming the token distance indicator based on the distance matrix; The passive token with the highest local density is set as the maximum distance between the current passive token and other passive tokens, the local density of the current passive token and the token distance indicator are combined, the score of each passive token is evaluated using a score function, and the passive token with the highest score is taken as the cluster representative; Assign other passive tokens to the nearest cluster representative to obtain the category and cluster results for each token.
6. A method for identifying trust risk allocation tokens according to claim 5, characterized in that: The token distance indicator is calculated as follows: ; in, represents the distance indicator of token i, represents the local density of any token i, represents the local density of any token j, Representation Token The corresponding token features, Representation Token The corresponding token features, Represents the covariance matrix of a multidimensional random variable; The local density is calculated as follows: ; in, Representation Token of A close neighbor.
7. An identification system based on trust risk allocation token, characterized in that: The identification method according to any one of claims 1 to 6 comprises: A first feature extraction module is configured to obtain a video sequence from the perspective of the drone, input the video sequence into the Vision Transformer backbone network, and use 3D convolution to extract spatiotemporal features of the video sequence to obtain a spatiotemporal representation; A second feature extraction module is configured to perform feature extraction and category score prediction on the spatiotemporal features to obtain intermediate features and category scores of the intermediate features; A frequency domain analysis module configured to use a fast Fourier transform to perform frequency domain information analysis on the intermediate features to obtain a dynamic significance level score; a token classification module configured to classify tokens in the intermediate features into active tokens and passive tokens based on a dynamic significance level score; an uncertainty calculation module configured to calculate a prediction uncertainty score for each passive token through a Gumbel-Softmax function using evidence theory and Dirichlet distribution; a token optimization module configured to sort and filter the passive tokens according to the predicted uncertainty scores of the passive tokens, optimize the passive tokens, and obtain optimized passive tokens; The behavior recognition module is configured to use the Vision Transformer backbone network to extract features from active tokens and optimized passive tokens to obtain behaviors in video sequences.
8. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; Wherein, the processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the identification method according to any one of claims 1 to 6.
