Complex video motion reasoning method based on differentiable fuzzy allen interval algebra

Abstract

The application discloses a complex video motion reasoning method based on a differentiable fuzzy Allen interval algebra, and solves the problems of a large number of oscillations and false detections of a neural network model on a transition frame caused by a hard-cut boundary in the prior art. The training video segment is input into the network model, and a feature sequence in the training video segment is extracted through a neural network module; then, the probability of each time step in the feature sequence as a motion center and the Gaussian distribution parameters of potential motions in the training video segment are predicted through a pyramid network module; then, the Allen relation membership degrees between any two fuzzy motion intervals are obtained through a pair relation reasoning module in combination with a preset Gaussian difference operator and an integral overlap operator; then, a full-connection time sequence logic graph composed of all fuzzy motion intervals is constructed in a graph convolution network module; finally, the network model is trained, and a test video segment is input into the trained model, so that the motion of the test video segment can be understood.

Description

Technical Field

[0001] This invention relates to artificial intelligence and computer vision technology, specifically to a method for structured modeling and reasoning of long-term complex events in videos using fuzzy logic. Background Technology

[0002] Video action understanding is a technology that uses computer vision to identify the types, temporal logic, and semantic meaning of human / object actions in videos. Its core is "understanding what the action is, when it happens, and what its significance is." With the rapid development of artificial intelligence and computer vision technology, video action understanding has evolved from simple classification tasks to temporal action localization (TAL) and the recognition of complex activities.

[0003] The current mainstream approach is to use bounding box (BBox) regression techniques in a sequence modeling neural network architecture (Transformer) based on self-attention mechanisms to forcibly predict the start and end timestamps of actions. However, real actions often occur gradually (e.g., "from sitting down to standing up"), and hard truncation of boundaries can easily cause the neural network model to generate a large number of oscillations and false detections on transition frames.

[0004] Furthermore, complex activities typically consist of multiple atomic actions; for example, "robbery" usually involves the sequence <approach, then, threaten, subsequently, seize>. While sequence modeling neural network architectures based on self-attention can capture long-distance dependencies, they learn the correlations between features rather than explicit topological structures. For instance, they cannot distinguish between "A before B" and "A after B," as these two are fundamentally different in terms of logical truth. Therefore, it is evident that this method suffers from a significant lack of topological logic. Summary of the Invention

[0005] The purpose of this invention is to provide a complex video action reasoning method based on differentiable fuzzy Allen interval algebra, which solves the technical problems of existing video action understanding methods, such as the large number of oscillations and false detections in the neural network model on transition frames due to hard truncation of boundaries, and the lack of topological logic.

[0006] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0007] A complex video action reasoning method based on differentiable fuzzy Allen interval algebra, characterized by the following steps:

[0008] Step 1: Prepare training video clips and test video clips; the content of the training video clips and test video clips are different;

[0009] Step 2: Construct the network model; the network model includes a neural network module, a pyramid network module, a pairwise relation reasoning module, and a graph convolutional network module connected in sequence; the neural network module is a pre-trained neural network;

[0010] Step 3: Input the training video clips into the neural network module, extract the feature sequences containing temporal context information from the training video clips, and output them to the pyramid network module;

[0011] Step 4: Predict the probability of each time step in the feature sequence being the action center, and the Gaussian distribution parameters of the potential actions in the training video segment through the pyramid network module; the Gaussian distribution parameters include all fuzzy action intervals, as well as the time center point and fuzzy duration half-width of each fuzzy action interval;

[0012] Step 5: Input any two fuzzy action intervals from all fuzzy action intervals into the pairwise relation reasoning module in sequence. Using the preset Gaussian difference operator and integral overlap operator, combined with the corresponding time center point and fuzzy duration half-width, calculate the Allen relation membership degree between any two fuzzy action intervals respectively.

[0013] Step 6: Using the semantic features of each fuzzy action interval as nodes and the membership degree of the Allen relation between any two fuzzy action intervals as the edge weights, construct a fully connected temporal logic graph composed of all fuzzy action intervals in the graph convolutional network module; the fully connected temporal logic graph includes a fully connected temporal logic structure and the feature vector of each node in the structure;

[0014] Step 7: Construct a loss function and iteratively train the network model using the methods from Steps 3 to 6 until the loss function converges to obtain the video action inference model;

[0015] Step 8: Input the test video clip described in Step 1 into the video action reasoning model, and output each action in the test video clip and its start and end times, thereby completing the complex video action reasoning based on differentiable fuzzy Allen interval algebra.

[0016] Furthermore, in step 2, the neural network module is a pre-trained 3D convolutional neural network or a video Transformer backbone network;

[0017] The pyramid network module is a one-dimensional temporal pyramid network. Each layer of the one-dimensional temporal pyramid network has a convolutional head with shared weights, and each convolutional head includes two parallel 1×1 convolutional layers.

[0018] Furthermore, in step 4, the expression for the time center point of the fuzzy action interval is as follows:

[0019] ;

[0020] In the formula, The time center point of the fuzzy action interval; This represents the anchor point position at the current time step in the feature sequence. This represents the step size of the current layer in the one-dimensional temporal pyramid network. It is the offset scaling factor of the time center point; It is the predicted offset of the time center point;

[0021] The expression for the fuzzy duration half-width of the fuzzy action interval is as follows:

[0022] ;

[0023] In the formula, It is the fuzzy duration half-width of the fuzzy action range. It is the prediction offset of the fuzzy continuous half-width.

[0024] Furthermore, in step 6, the feature vector of each node in the fully connected sequential logic structure is expressed by the following formula:

[0025] ;

[0026] In the formula, It is the feature vector of node i in the next layer after graph convolution in the current layer in a fully connected temporal logic structure; is the convolutional layer for the current index; ReLU is the activation function; r is the temporal relation index; Represents a set of temporal relationships; It is a learnable weight matrix for the time-series relation r; It is the feature vector of node j in the current layer; Let be the value at coordinate (i, j, r) in tensor A, where A is the edge weight tensor, and , It is the field of real numbers. This represents the total number of nodes, where i, j ∈ {1, 2, ..., N}. ; , It is a fuzzy membership function. This represents the fuzzy action range of node i. This represents the fuzzy action range of node j.

[0027] Furthermore, in step 7, the loss function is a joint loss function. Its expression is as follows:

[0028] ;

[0029] In the formula, It is the regression equilibrium coefficient. It is the logical balance coefficient; It is the loss used for action classification; It is the crossover ratio loss used to regress the parameters of a Gaussian distribution; This represents the loss due to topological consistency.

[0030] Further, in step 7, the... The expression is:

[0031] ;

[0032] In the formula, p is the probability that each time step in the feature sequence is the action center.

[0033] Further, in step 7, the... The expression is:

[0034] ;

[0035] In the formula, M is the normalization factor, Cost is the cost of a single violation, and

[0036] ;

[0037] in, express What happened Previous confidence level, Let k represent the fuzzy action range of node k, where k∈{1,2,…,N}.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. This invention inputs training video clips into a network model. The neural network module extracts feature sequences containing temporal context information from the training video clips. A pyramid network module then predicts the probability of each time step in the feature sequence being an action center and the Gaussian distribution parameters of potential actions in the training video clips. A pairwise relation reasoning module, combining preset Gaussian difference operators and integral overlap operators, obtains the Allen relation membership degree between any two fuzzy action intervals. Subsequently, a fully connected temporal logic graph composed of all fuzzy action intervals is constructed in a graph convolutional network module. The network model is iteratively trained using the training video clips until the loss function converges. Finally, a test video clip is input into the trained model to output each action in the test video clip and its start and end times. This invention transforms discrete symbolic logic into continuously differentiable mathematical operators, achieving end-to-end joint training of deep learning and logical reasoning. Through the deep fusion of qualitative logical knowledge and quantitative visual features, it significantly improves the video action reasoning model's ability to understand complex long videos and its boundary localization accuracy. It effectively solves the problem of numerous oscillations and false detections in transition frames caused by hard truncation boundaries in existing technologies.

[0040] 2. This invention models actions using Gaussian distribution parameters, transforming actions in video clips into fuzzy action intervals on the time axis, rather than definite start and end points. This method not only improves the robustness of detection but also provides downstream tasks with the key information of "uncertainty," realizing a paradigm shift from "hard regression" to "soft distribution."

[0041] 3. This invention transforms discrete logical relations into continuous Allen relation membership degrees by using a preset Gaussian difference operator and integral overlap operator, enabling the network model to learn logical rules using gradient descent, thereby realizing the "differentiability" of symbolic logic.

[0042] 4. This invention inputs test video clips into the video action reasoning model and explicitly uses structured information such as "A precedes B" and "A is contained by B" to assist in classification, thereby improving the realism of video action understanding. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating an embodiment of the complex video action reasoning method based on differentiable fuzzy Allen interval algebra of the present invention.

[0044] Figure 2The diagram shows the principle of the differentiable Allen relation operator in step 5 of the embodiment of the complex video action reasoning method based on differentiable fuzzy Allen interval algebra of the present invention; wherein (a) is the principle diagram of the Before relation operator; (b) is the principle diagram of the Overlaps relation operator; and (c) is the principle diagram of the During relation operator.

[0045] Figure 3 This is a fully connected temporal logic diagram (a) obtained through the video action reasoning model in step 8 of the embodiment of the complex video action reasoning method based on differentiable fuzzy Allen interval algebra of the present invention, and a schematic diagram (b) of the process of defuzzifying the fuzzy action interval and suppressing fuzzy nonmaxima.

[0046] Figure 4 The figures are a comparison of the temporal features of real actions, the hard regression results of existing technologies, and the inference results of this embodiment; wherein: (a) the figure is a temporal feature diagram of real actions without any processing; (b) the figure is a schematic diagram of the hard regression results of existing technologies; and (c) the figure is a schematic diagram of the inference results of this embodiment. Detailed Implementation

[0047] This invention aims to address two core problems in existing video action detection technologies: first, the difficulty in adapting to the gradual and ambiguous nature of action occurrence by regressing hard time boundaries (Start / End); and second, the lack of explicit modeling and reasoning capabilities for complex temporal logic between actions (such as "overlap" and "containment"). This invention achieves end-to-end joint training of deep learning and logical reasoning by transforming discrete symbolic logic into continuously differentiable mathematical operators.

[0048] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0049] like Figure 1 As shown, this embodiment provides a method for complex video action reasoning based on differentiable fuzzy Allen interval algebra, including the following steps:

[0050] Step 1: Prepare training video clips and test video clips; the content of the training video clips and test video clips are different.

[0051] Step 2: Construct the network model; the network model includes a neural network module, a pyramid network module, a pairwise relation reasoning module, and a graph convolutional network module connected in sequence.

[0052] Step 3: Input the training video clips into the neural network module, extract the feature sequences containing temporal context information from the training video clips, and output them to the pyramid network module.

[0053] The neural network module can be a pre-trained 3D convolutional neural network or a video Transformer backbone network. In this embodiment, a pre-trained I3D (Inception-3D) is selected as the backbone network. The input training video clips are resampled into Clip sequences. After I3D processing, the resolution can be maintained in the time dimension, and the corresponding feature sequence X is output.

[0054] Step 4: Predict the probability p of each time step in the feature sequence X as the action center using the pyramid network module, and train the Gaussian distribution parameters of the potential actions in the video clip, thereby modeling the potential actions as a Gaussian fuzzy set on the time axis. The Gaussian distribution parameters include all fuzzy action intervals, as well as the time center point and fuzzy duration half-width of each fuzzy action interval.

[0055] To capture actions at different scales and output multi-scale feature maps, this embodiment adds a pyramid network module after the backbone network. The pyramid network module is a one-dimensional temporal pyramid network (1D-FPN). Each layer of the one-dimensional temporal pyramid network has a convolutional head with shared weights, and each convolutional head includes two parallel 1×1 convolutional layers. One of these is a classification branch, used to output the probability that each time step is the center of an action. This probability is expressed as the confidence that an action exists at that time step. This is reflected in the first branch. The second branch is a regression branch, which outputs the Gaussian distribution parameters of the potential actions in the training video clips.

[0056] The expression for the time center point of the fuzzy action interval described in this embodiment is as follows:

[0057] ;

[0058] In the formula, The time center point of the fuzzy action interval; This represents the anchor point position at the current time step in the feature sequence. This represents the step size of the current layer in the one-dimensional temporal pyramid network. It is the offset scaling factor of the time center point; It is the predicted offset of the time center point, which is directly output by the regression branch; It is the hyperbolic tangent activation function.

[0059] The expression for the fuzzy duration half-width of the fuzzy action interval is as follows:

[0060] ;

[0061] In the formula, It is the fuzzy duration half-width of the fuzzy action range. It is the prediction offset of the fuzzy continuous half-width, which is directly output by the regression branch.

[0062] Note that: An exponential function or the Softplus activation function should be used to ensure... It is always a positive value.

[0063] Step 5: Input any two fuzzy action intervals from all fuzzy action intervals into the pairwise relation reasoning module in sequence. Using the preset Gaussian difference operator and integral overlap operator, combined with the corresponding time center point and fuzzy duration half-width, calculate the Allen relation membership degree between any two fuzzy action intervals.

[0064] The Allen relation membership degree described in this embodiment is a fuzzy membership degree that satisfies the 13 basic temporal relations (such as Before, Overlaps, During, etc.) defined in Allen interval algebra, which transforms discrete logical rules into a continuous mathematical calculation process that can be used for gradient descent training.

[0065] The following uses the Before, Overlaps, and During relational operators as examples to calculate the fuzzy action interval. , Logical relationship between them:

[0066] (A) Before relational operator ( exist Before)

[0067] Logical meaning: The tail section is clearly earlier than The head, such as Figure 2 As shown in Figure (a), This represents the true distance between the time centers of two fuzzy action intervals. This represents the predicted distance between the time centers of two fuzzy action intervals, when hour, .

[0068] Mathematical definition:

[0069]

[0070] in, It is a fuzzy membership degree, representing an action. What happened Previous confidence level; It's the temperature coefficient, used for control logic judgment; It is the time center of the preceding fuzzy action interval; It is the fuzzy duration half-width of the preceding fuzzy action interval; It is the time center of the subsequent fuzzy action interval; It is the fuzzy duration half-width of the subsequent fuzzy action interval; It is an activation function.

[0071] (B) Overlaps relational operator ( and overlapping)

[0072] Logical meaning: and There is a significant area of ​​intersection on the time axis, such as Figure 2 As shown in Figure (b) of the document.

[0073] Mathematical definition: Approximated using Gaussian product integral formula:

[0074]

[0075] The formula has a value of 1 when the two distributions completely overlap, and the value decreases as the separation increases. It is a fuzzy membership degree, representing an action. and Overlapping confidence levels.

[0076] (C) During relational operators ( quilt Include)

[0077] Logical meaning: (e.g., chopping vegetables) The duration is shorter than (e.g., cooking) and the centers of both are close to each other, that is ,like Figure 2 As shown in Figure (c).

[0078] Mathematical definition:

[0079]

[0080] It is a fuzzy membership degree, representing Include The confidence level.

[0081] Step 6: Using the semantic features of each fuzzy action interval as nodes and the membership degree of the Allen relation between any two fuzzy action intervals as the edge weights, construct a fully connected temporal logic graph composed of all fuzzy action intervals in the Graph Convolutional Network (GCN) module.

[0082] The graph convolutional network module in this embodiment is a fuzzy temporal graph convolutional network (Fuzzy Temporal GCN). The fully connected temporal logic graph includes a fully connected temporal logic structure and the feature vector of each node in the structure. The feature vector of each node is expressed by the following formula:

[0083]

[0084] In the formula, It is the feature vector of node i in the next layer after graph convolution in the current layer in a fully connected temporal logic structure; is the convolutional layer for the current index; ReLU is the activation function; r is the temporal relation index; Represents a set of temporal relationships; It is a learnable weight matrix for the time-series relation r; It is the feature vector of node j in the current layer; Let be the value at coordinate (i, j, r) in tensor A, where A is the edge weight tensor, and , It is the field of real numbers. This represents the total number of nodes, where i, j ∈ {1, 2, ..., N}. ; , It is a fuzzy membership function. This represents the fuzzy action range of node i. This represents the fuzzy action range of node j.

[0085] Step 7: Construct a loss function and iteratively train the network model using the methods from Steps 3 to 6 until the loss function converges, thus obtaining the video action inference model.

[0086] The loss function described in this embodiment is a joint loss function. Its expression is as follows:

[0087]

[0088] In the formula, It is the regression equilibrium coefficient. It is the logical balance coefficient; It is a Focal Loss used for action classification. The expression is:

[0089]

[0090] In the formula, p is the probability that each time step in the feature sequence is the action center.

[0091] It is the crossover ratio loss (IoU Loss) used to regress the parameters of a Gaussian distribution.

[0092] Topological consistency loss is used to force the relationships learned by the network model to conform to logical axioms (transitivity) for any triplet. ,if Before and Before ,but Must Before ,but:

[0093]

[0094] Cost is the penalty for a single violation. express What happened Previous confidence level, Let k ∈ {1, 2, ..., N} be the fuzzy action range of node k. The expression is:

[0095]

[0096] In the formula, M is the normalization factor.

[0097] Step 8: Input the test video clip described in Step 1 into the video action inference model, and output each action in the test video clip (i.e., achieve action classification) and its start and end times (i.e., obtained through deblurring and boundary conditions). and This allows for the completion of complex video action reasoning based on differentiable fuzzy Allen interval algebra.

[0098] In this inference phase, the feature vector of the action is updated according to the fully connected temporal logic structure, defuzzification decoding is performed using fuzzy parameters, and the precise start and end times of the action are output. Finally, redundant results are removed through fuzzy nonmaximum suppression based on KL divergence (Kullback-Leibler Divergence). Specifically:

[0099] 1. Construct a fully connected sequential logic diagram

[0100] First, the test video clip is input into the video action reasoning model. The pre-trained neural network module extracts the feature sequence containing temporal context information from the test video clip and outputs it to the pyramid network module. The pyramid network module predicts the probability of each time step in the feature sequence being the action center and the Gaussian distribution parameters of the potential actions in the training video clip. The pairwise relation reasoning module uses the preset Gaussian difference operator and integral overlap operator combined with the corresponding Gaussian distribution parameters to calculate the Allen relation membership degree between any two fuzzy action intervals. Finally, a fully connected temporal logic graph composed of all fuzzy action intervals is constructed through the graph convolutional network module.

[0101] like Figure 3 As shown in Figure (a) on the left, when , When there is a 90% certainty, action A connects to action B, meaning action B begins the instant action A ends; when , When there is an 80% certainty, action B overlaps with action C, and action B starts and ends first.

[0102] 2. Defuzzification and Non-Maximum Suppression (NMS) processing

[0103] like Figure 3 As shown in Figure (b) on the right, the core of defuzzification is to transform the predicted Gaussian distribution parameters into deterministic start and end time boundaries of any two fuzzy action intervals, i.e. (corresponding to the 95% confidence interval). It is the start time of the action.

[0104] To remove redundant bounding boxes, IoU Loss is no longer used; instead, the KL divergence between the two blurred action intervals is calculated. If both describe the same action, then the one with lower confidence should be suppressed; if If the two statements describe different actions, then they should be output separately. , This indicates two actions. It is a preset overlap threshold.

[0105] like Figure 4As shown, (a) Figure shows the temporal characteristics of a real action without any processing, which has a real temporal gradual change process; (b) Figure shows the hard regression method of the prior art, which hard truncates the boundary, so the inferred action has boundary error; (c) Figure shows the inference method of this embodiment. As can be seen from the figure, the inference result of this embodiment has a high degree of consistency with the temporal characteristics of the real action, and its inference result is closer to the real action.

[0106] This embodiment introduces Allen's Interval Algebra, a classic logic theory that describes relationships between time intervals. This invention innovatively "fuzzifies" and "neuralizes" this theory. By defining fuzzy time intervals, the calculation of logical relationships (such as overlaps and meetets) becomes continuously differentiable, thereby enabling the embedding of deep networks for joint optimization.

[0107] This invention is particularly suitable for scenarios that require precise positioning of action boundaries and understanding of the logical relationships between actions, such as fighting detection in intelligent security (distinguishing between the temporal evolution of "pushing" and "mutual fighting"), standardization detection of industrial assembly line operations, and tactical analysis of sports competition videos.

[0108] In addition, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the complex video action reasoning method based on differentiable fuzzy Allen interval algebra described in this embodiment.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein, and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A method for complex video action reasoning based on differentiable fuzzy Allen interval algebra, characterized in that, Includes the following steps: Step 1: Prepare training video clips and test video clips; the content of the training video clips and test video clips are different; Step 2: Construct the network model; the network model includes a neural network module, a pyramid network module, a pairwise relation reasoning module, and a graph convolutional network module connected in sequence; the neural network module is a pre-trained neural network; Step 3: Input the training video clips into the neural network module, extract the feature sequences containing temporal context information from the training video clips, and output them to the pyramid network module; Step 4: Predict the probability of each time step in the feature sequence being the action center, and the Gaussian distribution parameters of the potential actions in the training video segment through the pyramid network module; the Gaussian distribution parameters include all fuzzy action intervals, as well as the time center point and fuzzy duration half-width of each fuzzy action interval; Step 5: Input any two fuzzy action intervals from all fuzzy action intervals into the pairwise relation reasoning module in sequence. Using the preset Gaussian difference operator and integral overlap operator, combined with the corresponding time center point and fuzzy duration half-width, calculate the Allen relation membership degree between any two fuzzy action intervals respectively. Step 6: Using the semantic features of each fuzzy action interval as nodes and the membership degree of the Allen relation between any two fuzzy action intervals as the edge weights, construct a fully connected temporal logic graph composed of all fuzzy action intervals in the graph convolutional network module; the fully connected temporal logic graph includes a fully connected temporal logic structure and the feature vector of each node in the structure; Step 7: Construct a loss function and iteratively train the network model using the methods from Steps 3 to 6 until the loss function converges to obtain the video action inference model; Step 8: Input the test video clip described in Step 1 into the video action reasoning model, and output each action in the test video clip and its start and end times, thereby completing the complex video action reasoning based on differentiable fuzzy Allen interval algebra.

2. The complex video action reasoning method based on differentiable fuzzy Allen interval algebra according to claim 1, characterized in that: In step 2, the neural network module is a pre-trained 3D convolutional neural network or a video Transformer backbone network; The pyramid network module is a one-dimensional temporal pyramid network. Each layer of the one-dimensional temporal pyramid network has a convolutional head with shared weights, and each convolutional head includes two parallel 1×1 convolutional layers.

3. The complex video action reasoning method based on differentiable fuzzy Allen interval algebra according to claim 1, characterized in that: In step 4, the expression for the time center point of the fuzzy action interval is as follows: ； In the formula, The time center point of the fuzzy action interval; This represents the anchor point position at the current time step in the feature sequence. This represents the step size of the current layer in the one-dimensional temporal pyramid network. It is the offset scaling factor of the time center point; It is the predicted offset of the time center point; The expression for the fuzzy duration half-width of the fuzzy action interval is as follows: ； In the formula, It is the fuzzy duration half-width of the fuzzy action range. It is the prediction offset of the fuzzy continuous half-width.

4. The complex video action reasoning method based on differentiable fuzzy Allen interval algebra according to claim 3, characterized in that: In step 6, the feature vector of each node in the fully connected sequential logic structure is expressed by the following formula: ； In the formula, It is the feature vector of node i in the fully connected temporal logic structure in the next layer after graph convolution in the current layer; is the convolutional layer for the current index; ReLU is the activation function; r is the temporal relation index; Represents a set of temporal relationships; It is a learnable weight matrix for the time-series relation r; It is the feature vector of node j in the current layer; Let be the value at coordinate (i, j, r) in tensor A, where A is the edge weight tensor, and , It is the field of real numbers. This represents the total number of nodes, where i, j ∈ {1, 2, ..., N}. ; , It is a fuzzy membership function. This represents the fuzzy action range of node i. This represents the fuzzy action range of node j.

5. The complex video action reasoning method based on differentiable fuzzy Allen interval algebra according to claim 4, characterized in that: In step 7, the loss function is a joint loss function. Its expression is as follows: ； In the formula, It is the regression equilibrium coefficient. It is the logical balance coefficient; It is the loss used for action classification; It is the crossover ratio loss used to regress the parameters of a Gaussian distribution; This represents the loss due to topological consistency.

6. The complex video action reasoning method based on differentiable fuzzy Allen interval algebra according to claim 5, characterized in that: In step 7, the The expression is: ； In the formula, p is the probability that each time step in the feature sequence is the action center.

7. The complex video action reasoning method based on differentiable fuzzy Allen interval algebra according to claim 5, characterized in that: In step 7, the The expression is: ； In the formula, M is the normalization factor, Cost is the cost of a single violation, and ； in, express What happened Previous confidence level, Let k represent the fuzzy action range of node k, where k∈{1,2,…,N}.

Images