Long-term activity analysis method and system based on hierarchical predefined knowledge learning

Abstract

The present invention provides a method and system for long-term activity analysis based on hierarchical predefined knowledge learning, the method comprising: utilizing multimodal semantic consistency, embedding the video clips of tutorial activities and their corresponding text descriptions into a joint semantic space through comparative learning, and learning the prior distribution of the joint representation of visual language; sampling some video text pair samples from the downstream data set in proportion, clustering their representations, and constructing a corresponding knowledge graph for different granularity levels; forming a systematic predefined knowledge graph; finding the concept node with the highest matching degree in each layer of the knowledge graph, fusing the representation of the concept node through a self-attention fusion strategy, and then fusing it with the original representation of the sample of the given modality to obtain an enhanced feature representation of the sample. The advantages of the present invention are that the proposed systematic predefined knowledge enhancement method achieves the best performance in the behavior segmentation task, and the frame-level accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video behavior recognition, and in particular to a method for analyzing long-term activities in multimodal tutorial videos. Background Art

[0002] Tutorial videos are videos that teach step-by-step instructions for complex daily tasks, such as cooking and changing tires. With the popularity of portable video capture devices and short video sharing platforms, a large number of tutorial videos have emerged on the Internet. Tutorial activity videos have their own characteristics. Compared with ordinary action videos, they focus on long-time steps of a few seconds to 1 minute. They usually have video-text step pairs, with weak dependencies such as sequence and causality between steps. The same video contains different sub-steps of the same task. Early tutorial activity analysis datasets were mainly manually annotated. Although the existing manual annotation methods have concise and clear step definitions and accurately annotated time activity boundaries, these tutorial activity videos are extremely time-consuming and costly, and difficult to scale up on a large scale. Recently, many ultra-large-scale multimodal tutorial video activity analysis datasets have emerged. These datasets are huge in scale, have a full range of activities, and contain weakly aligned video-text pairs. Summary of the invention

[0003] The technical problem to be solved by the present invention is how to reduce the dependence on manually labeled samples and improve the data utilization rate of the model.

[0004] The present invention solves the above technical problems through the following technical means:

[0005] The present invention discloses a long-term activity analysis method based on hierarchical predefined knowledge learning, comprising the following steps:

[0006] S1. Using multimodal semantic consistency, we embed the video clips of tutorial activities and their corresponding text descriptions into a joint semantic space through contrastive learning, and learn the prior distribution of the joint representation of vision and language.

[0007] S2. Sample some video text pairs from the downstream data set in proportion, cluster their representations, and obtain hierarchical sample clusters from top to bottom to achieve the construction of hierarchical concept nodes; for different granularity levels, construct a corresponding knowledge graph;

[0008] S3. Through deep random path exploration, random walk sequences with fixed walk lengths are sampled. The obtained random walk sequences are regarded as virtual sentences, and the nodes in the random walk sequences are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted to relax the optimization problem. The attribute information of all nodes in the hierarchical knowledge graph is updated to form the final systematic predefined knowledge graph.

[0009] S4. Calculate the similarity between the predefined knowledge representation of the downstream query sample of the given modality and the representation of the corresponding modality of all nodes in each layer of the systematic predefined knowledge graph to find the concept node with the highest matching degree in each layer of the knowledge graph; fuse the representation of the concept node through the self-attention fusion strategy; and then fuse it with the original representation of the sample of the given modality to obtain the enhanced feature representation of the sample.

[0010] Furthermore, the step S1 is specifically as follows:

[0011] S11. Using multimodal semantic consistency, we embed the video clips of tutorial activities and their corresponding text descriptions into a d-dimensional joint semantic space through contrastive learning to learn the prior distribution of the joint representation of vision and language. , where f, g are the video encoder network and text encoder network respectively, x, y are the input video clips and the corresponding text descriptions respectively;

[0012] S12. Construct a contrastive learning agent task and use the softmax version of the noise contrast estimation loss function to self-supervise and learn ;

[0013] S13, using the MIL-NCE loss function, relax a single positive sample pair into a candidate set of positive sample pairs , obtain better self-supervised visual-language joint representation priors on the ultra-large-scale tutorial video dataset HowTo100M.

[0014] The step S2 comprises the following steps:

[0015] S21. Constructing hierarchical concept nodes: from downstream datasets In proportion Sample some video text pairs and represent them Hierarchical k-Means algorithm is used for clustering, and hierarchical sample clusters are obtained from top to bottom. They are video representation and text representation respectively;

[0016] S22. Construct a hierarchical prior knowledge graph: For different levels of granularity m, construct a corresponding knowledge graph ,node By this layer Cluster Centers Composition, among which is the set of nodes of the m-th layer knowledge graph, is the set of edges of the m-th layer knowledge graph, are the cluster centers of the mth layer respectively The video representation and text representation in the , the similarity between any two concept nodes i and j is determined by the joint representation of the video and text of the cluster center. Considering the binary graph, the similarity is quantified according to the threshold to obtain the prior knowledge graphs of different granularities.

[0017] Furthermore, the step S21 is specifically as follows:

[0018] Set the number of clustering levels to , the branch parameters of each layer are , from top to bottom, each layer will get Hierarchical clustering. is the Mth layer branch parameter; each node in each layer is regarded as an abstract concept, representing the overall properties of all samples in the corresponding cluster.

[0019] The step S22 is specifically as follows:

[0020] The similarity between any two concept nodes i and j is determined by the joint representation of the video and text at the cluster center, namely:

[0021]

[0022] in is element-wise multiplication, is the video representation in the i-th cluster center of the m-th layer, is the video representation in the jth cluster center of the mth layer, is the text representation in the i-th cluster center of the m-th layer, is the text representation in the jth cluster center of the mth layer; considering the binary graph, according to the threshold Similarity Quantify and generate the edges of the knowledge graph , represents the relationship between concept nodes in the graph, namely:

[0023]

[0024] Where sgn is a symbolic function, and M layers of prior knowledge graphs of different granularities are obtained , represents the pre-trained video encoder network and the text encoder network For downstream datasets The hierarchical prior expression of It is the Mth layer knowledge graph.

[0025] Furthermore, in step S3, the random walk sequence with a fixed walk length is sampled by deep random path exploration as follows:

[0026] Use DeepWalk graph embedding technology to embed the m-th layer of knowledge graph The structural relationship in the hierarchical knowledge graph is learned and reasoned; for the mth layer of the knowledge graph , through random path exploration, a random walk sequence with a fixed walk length t is sampled.

[0027] In step S3, the obtained random walk sequence is regarded as a virtual sentence, and the nodes in the random walk sequence are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted. The specific relaxation optimization problem is:

[0028] Learn and reason about the m-th level knowledge graph Node embedding, that is, learning the following mapping function :

[0029]

[0030] in For nodes, is the mth level knowledge graph The collection of nodes, is the number of nodes, d is the dimension of node embedding;

[0031] The optimization agent task is set as follows: given a sequence of nodes that have been experienced in a random path exploration, predict a certain observation node The maximum likelihood that the next ,Right now:

[0032]

[0033] Represents the mth level knowledge graph The observation nodes in Through the mapping function Mapping is done.

[0034] Using missing words to predict the context, the optimization problem is relaxed to:

[0035]

[0036] The nodes The subscript indicates the observation node The position of the relevant nodes, the objective function models the length of 2 The transition probability of random path exploration is optimized, and the m-th layer knowledge graph The structural information is encoded into the embedding information of each abstract concept node in this layer. ;

[0037] Update the attribute information of all nodes in the hierarchical knowledge graph to form the final systematic predefined knowledge graph:

[0038] Hierarchical Knowledge Graph The attribute information of all nodes is updated to ,in For all original attribute information, Embed all new systematized information to form the final systematized predefined knowledge graph .

[0039] The present invention also provides a long-term activity analysis system based on hierarchical predefined knowledge learning, comprising the following modules:

[0040] An unstructured self-supervised representation prior building module that exploits multimodal semantic consistency and embeds video clips of tutorial activities and their corresponding text descriptions into a joint semantic space through contrastive learning to learn a joint visual-linguistic representation prior distribution.

[0041] A systematic predefined knowledge graph construction module is used to sample some video text pairs from the downstream data set in proportion, cluster their representations, and obtain hierarchical sample clustering clusters from top to bottom to realize the construction of hierarchical concept nodes; for different granularity levels, a corresponding knowledge graph is constructed;

[0042] The learning and reasoning module of systematic predefined knowledge is used to sample random walk sequences of fixed walk lengths through deep random path exploration. The obtained random walk sequences are regarded as virtual sentences, and the nodes in the random walk sequences are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted to relax the optimization problem. The attribute information of all nodes in the hierarchical knowledge graph is updated to form the final systematic predefined knowledge graph.

[0043] The downstream task performance enhancement module is used to calculate the similarity between the predefined knowledge representation of the downstream query sample of the given modality and the representation of the corresponding modality of all nodes in each layer of the systematic predefined knowledge graph, and find the concept node with the highest matching degree in each layer of the knowledge graph; fuse the representation of the concept node through the self-attention fusion strategy; and then fuse it with the original representation of the sample of the given modality to obtain the enhanced feature representation of the sample.

[0044] Furthermore, the non-systematic self-supervised representation prior construction module is used to utilize multimodal semantic consistency to embed the video clips of the tutorial activity and their corresponding text descriptions into a joint semantic space through comparative learning, and learn the visual language joint representation prior distribution specifically as follows:

[0045] Using multimodal semantic consistency, we embed the video clips of tutorial activities and their corresponding text descriptions into a d-dimensional joint semantic space through contrastive learning to learn the prior distribution of the joint visual-linguistic representation. , where f, g are the video encoder network and text encoder network respectively, x, y are the input video clips and the corresponding text descriptions respectively;

[0046] Construct a contrastive learning agent task and use the softmax version of the noise contrast estimation loss function to self-supervise and learn ;

[0047] Using the MIL-NCE loss function, a single positive sample pair is relaxed into a candidate set of positive sample pairs. ,Obtaining better self-supervised visual-language joint representation priors on the tutorial video dataset.

[0048] In the learning and reasoning module of the systematic predefined knowledge, a random walk sequence with a fixed walk length is sampled by deep random path exploration as follows:

[0049] Use DeepWalk graph embedding technology to embed the mth knowledge graph The structural relationship in the hierarchical knowledge graph is learned and reasoned; for the mth layer of the knowledge graph , through random path exploration, a random walk sequence with a fixed walk length t is sampled.

[0050] The obtained random walk sequence is regarded as a virtual sentence, and the nodes in the random walk sequence are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted. The specific relaxation optimization problem is:

[0051] Learn and reason about the m-th level knowledge graph Node embedding, that is, learning the following mapping function :

[0052]

[0053] in For nodes, is the mth level knowledge graph The collection of nodes, is the number of nodes, d is the dimension of node embedding;

[0054] The optimization agent task is set as follows: given a sequence of nodes that have been experienced in a random path exploration, predict a certain observation node The maximum likelihood that the next ,Right now:

[0055]

[0056] Represents the mth level knowledge graph The observation nodes in Through the mapping function Mapping is done.

[0057] Using missing words to predict the context, the optimization problem is relaxed to:

[0058]

[0059] The nodes The subscript indicates the observation node The position of the relevant nodes, the objective function models the length of 2 The transition probability of random path exploration is optimized, and the m-th layer knowledge graph The structural information is encoded into the embedding information of each abstract concept node in this layer. ;

[0060] Update the attribute information of all nodes in the hierarchical knowledge graph to form the final systematic predefined knowledge graph:

[0061] Hierarchical Knowledge Graph The attribute information of all nodes is updated to ,in For all original attribute information, Embed all new systematized information to form the final systematized predefined knowledge graph .

[0062] The advantages of the present invention are:

[0063] (1) The proposed systematic predefined knowledge augmentation method achieves the best performance on the behavior segmentation task and improves the frame-level accuracy.

[0064] (2) In the text-to-video retrieval task, it achieves the best performance in terms of recall rate and median recall ranking.

[0065] (3) In the task of sequential step localization, there is no need to explicitly introduce dataset-specific heuristic constraints. In the process of generating model step nominations, systematic predefined knowledge representation is used to improve the localization performance, and even better step localization performance is achieved than the method of fusing optical flow information. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of a long-term activity analysis method based on hierarchical predefined knowledge learning according to Embodiment 1 of the present invention;

[0067] Figure 2A schematic diagram of a priori self-supervised visual language joint representation in Example 1 of the present invention;

[0068] Figure 3 A schematic diagram of constructing a hierarchical knowledge graph and learning and reasoning of predefined knowledge in Example 1 of the present invention;

[0069] Figure 4 Schematic diagram of learning and reasoning of systematized predefined knowledge in Example 1 of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Example 1

[0072] The present invention discloses a long-term activity analysis method based on hierarchical predefined knowledge learning, such as Figure 1 As shown, the method comprises the following steps:

[0073] S1. Using multimodal semantic consistency, we embed the video clips of tutorial activities and their corresponding text descriptions into a joint semantic space through contrastive learning, and learn the prior distribution of the joint representation of vision and language, such as Figure 2 As shown, the automatically generated annotated multimodal tutorial video naturally has weakly aligned video and text pairs. The step S1 is specifically as follows:

[0074] S11. Self-supervised translation of video clips of tutorial activities through contrastive learning using multimodal semantic consistency The corresponding text description Embedded into a d-dimensional joint semantic space , learn the prior distribution of the joint representation of vision and language , where f and g are Figure 2 The video encoder network and text encoder network in are, x, y are the input video clips and the corresponding text descriptions respectively;

[0075] S12. Construct a contrastive learning agent task and use the softmax version of the noise contrast estimation loss function to self-supervise and learn ;

[0076] S13, using the MIL-NCE loss function, relax a single positive sample pair into a candidate set of positive sample pairs ,Right now:

[0077]

[0078] Where f and g are Figure 2 The video encoder network and text encoder network in ; x, y are the input video clip and the corresponding text description respectively; is the candidate set of negative samples, n is the number of samples, and T is the matrix transpose symbol, because , so it is defined , then the meaning of x', y' is clear.

[0079] Obtaining Better Self-Supervised Visual-Language Joint Representation Priors on the Very Large-Scale Tutorial Video Dataset HowTo100M.

[0080] S2. Sample some video text pairs in proportion from the downstream data set, cluster their representations, and obtain hierarchical sample clustering clusters from top to bottom to realize the construction of hierarchical concept nodes; for different granularity levels, construct a corresponding knowledge graph, and the step S2 includes the following steps:

[0081] S21. Constructing hierarchical concept nodes: from downstream datasets Some video text pairs are sampled in proportion and their representations are Hierarchical k-Means algorithm is used for clustering, and hierarchical sample clusters are obtained from top to bottom. are video representation and text representation respectively. If the number of clustering levels is set to , the branch parameters of each layer are , from top to bottom, each layer will get Hierarchical clustering. is the M-th layer branch parameter. Figure 3 As shown, In the clustering process, each node in each layer is regarded as an abstract concept, which represents the overall properties of all samples in the corresponding cluster.

[0082] S22. Construct a hierarchical prior knowledge graph: For different levels of granularity m, construct a corresponding knowledge graph ,node By this layer Cluster Centers Composition, among which is the set of nodes of the m-th layer knowledge graph, is the set of edges of the m-th layer knowledge graph, are the cluster centers of the mth layer respectively The video representation and text representation in , the similarity between any two concept nodes i and j is determined by the joint representation of the video and text of the cluster center, that is:

[0083]

[0084] in is element-wise multiplication, is the video representation in the i-th cluster center of the m-th layer, is the video representation in the jth cluster center of the mth layer, is the text representation in the i-th cluster center of the m-th layer, is the text representation in the jth cluster center of the mth layer. To simplify the problem, only the binary graph is considered here, so according to the threshold Similarity Quantify to generate the edge of the knowledge graph , which represents the relationship between concept nodes in the graph, namely:

[0085]

[0086] Where sgn is a symbolic function, and eventually we will get M layers of prior knowledge graphs of different granularities. , represents the prior model (i.e., the pre-trained video encoder network and the text encoder network ) for downstream datasets The hierarchical prior expression of It is the Mth layer knowledge graph.

[0087] S3. Through deep random path exploration, random walk sequences with fixed walk lengths are sampled; the obtained random walk sequences are regarded as virtual sentences, and the nodes in the random walk sequences are regarded as virtual words, and the node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted to relax the optimization problem; the attribute information of all nodes in the hierarchical knowledge graph is updated to form the final systematic predefined knowledge graph, such as Figure 4 As shown, step S3 includes the following steps:

[0088] S31. Through deep random path exploration, a random walk sequence with a fixed walk length is sampled. Specifically, the DeepWalk graph embedding technology is used to embed the m-th layer of knowledge graph. The structural relationship in the hierarchical knowledge graph is learned and reasoned. , through random path exploration, a random walk sequence with a fixed walk length t is sampled.

[0089] S32: The obtained random walk sequence is regarded as a virtual sentence, and the nodes in the random walk sequence are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted to relax the optimization problem. The specific S32 is: learning and inferring the mth layer of knowledge graph Node embedding, that is, learning the following mapping function :

[0090]

[0091] in For nodes, is the mth level knowledge graph The collection of nodes, is the number of nodes, and d is the dimension of node embedding.

[0092] The optimized agent task is: given a sequence of nodes that have been experienced in a random path exploration, predict a certain observation node The maximum likelihood that the next ,Right now:

[0093]

[0094] Represents the mth level knowledge graph The observation nodes in Through the mapping function Mapping is done.

[0095] However, because the random walk sequence has sequential dependencies, as the length of the path exploration increases, the calculation of the probability becomes very difficult. Therefore, the optimization problem can be relaxed to:

[0096]

[0097] The nodes The subscript indicates the observation node The position of the relevant nodes, the objective function models the length of 2 The transition probability of random path exploration is optimized, and the m-th layer knowledge graph The structural information is encoded into the embedding information of each abstract concept node in this layer. .

[0098] S33, updating the attribute information of all nodes in the hierarchical knowledge graph to form the final systematic predefined knowledge graph. The step S33 is specifically as follows: and new systematized embedded information In this embodiment, the hierarchical knowledge graph The attribute information of all nodes is updated to ,in For all original attribute information, Embed all new systematized information to form the final systematized predefined knowledge graph , used to assist downstream small-scale tutorial video datasets The specific tutorial long-term activity analysis task, the specific process is as follows Figure 4 shown.

[0099] S4. Predefined knowledge representation and systematized predefined knowledge graph of downstream query samples of given modality The similarity of the representations of the corresponding modalities of all nodes in each layer is calculated to find the concept nodes with the highest matching degree in each layer of the knowledge graph; the representations of the concept nodes are fused through the self-attention fusion strategy; and then fused with the original representation of the sample of the given modality to obtain the enhanced feature representation of the sample. The specific step S4 is:

[0100] First, by giving the modality Sample downstream query Predefined knowledge representation With systematic predefined knowledge graph Representation of the corresponding modes of all nodes in each layer Calculate similarity. Considering that each node representation already contains the systematic information of the knowledge graph, we only need to find the concept node with the highest matching degree in each layer of the knowledge graph. ,Right now:

[0101]

[0102] in Represents the mth level knowledge graph The i-th node in Represents the mth level knowledge graph The i-th node in corresponds to the representation of the mode, It can be various similarity calculation functions, and the simple cosine similarity is used here. Therefore, for the M-layer systematic predefined knowledge graph, a total of M most relevant concept nodes at different levels are matched. , and its corresponding representation These best matches represent Finally, the representations of these M concept nodes are fused through the self-attention fusion strategy. This mechanism can adaptively weight the feature representations of abstract concept nodes at different levels, which has stronger selectivity and adaptability and the best performance. After fusion, it is combined with the original representation of the sample q of the given modality. Fusion, to obtain the enhanced feature representation of sample q.

[0103] Example 2

[0104] Based on the same inventive concept of Example 1, the present invention further provides a long-term activity analysis system based on hierarchical predefined knowledge learning, the system comprising the following modules:

[0105] 1. A non-systematic self-supervised representation prior construction module is used to utilize multimodal semantic consistency and embed the video clips of tutorial activities and their corresponding text descriptions into a joint semantic space through comparative learning to learn the visual language joint representation prior distribution. Furthermore, the non-systematic self-supervised representation prior construction module operates in the following specific ways:

[0106] (1) Using multimodal semantic consistency, we self-supervise the video clips of tutorial activities through contrastive learning. The corresponding text description Embedded into a d-dimensional joint semantic space , learn the prior distribution of the joint representation of vision and language , where f and g are Figure 2 The video encoder network and text encoder network in are, x, y are the input video clips and the corresponding text descriptions respectively;

[0107] (2) Construct a contrastive learning agent task and use the softmax version of the noise contrast estimation loss function to self-supervise and learn .

[0108] (3) Using the MIL-NCE loss function, a single positive sample pair is relaxed into a candidate set of positive sample pairs ,Right now:

[0109]

[0110] Where f and g are Figure 2 The video encoder network and text encoder network in ; x, y are the input video clip and the corresponding text description respectively; is the candidate set of negative samples, n is the number of samples, and T is the matrix transpose symbol, because , so it is defined , then the meaning of x', y' is clear.

[0111] Obtaining Better Self-Supervised Visual-Language Joint Representation Priors on the Very Large-Scale Tutorial Video Dataset HowTo100M.

[0112] The systematized predefined knowledge graph construction module is used to sample some video text pairs from the downstream data set in proportion, cluster their representations, and obtain hierarchical sample clusters from top to bottom to achieve the construction of hierarchical concept nodes; for different granularity levels, a corresponding knowledge graph is constructed. Furthermore, the specific operation mode of the systematized predefined knowledge graph construction module is as follows: (1) Construct hierarchical concept nodes: In proportion Sample some video text pairs and represent them Hierarchical k-Means algorithm is used for clustering, and hierarchical sample clusters are obtained from top to bottom. are video representation and text representation respectively. Furthermore, if the number of clustering levels is set to , the branch parameters of each layer are , from top to bottom, each layer will get Hierarchical clustering. is the branch parameter of the Mth layer. Each node in each layer is regarded as an abstract concept, which represents the overall properties of all samples in the corresponding cluster.

[0113] (2) Constructing a hierarchical prior knowledge graph: For different levels of granularity m, construct a corresponding knowledge graph ,node By this layer Cluster Centers Composition, among which is the set of nodes of the m-th layer knowledge graph, is the set of edges of the m-th layer knowledge graph, are the cluster centers of the mth layer respectively The video representation and text representation in , the similarity between any two concept nodes i and j is determined by the joint representation of the video and text of the cluster center, that is:

[0114]

[0115] in is element-wise multiplication, is the video representation in the i-th cluster center of the m-th layer, is the video representation in the jth cluster center of the mth layer, is the text representation in the i-th cluster center of the m-th layer, is the text representation in the jth cluster center of the mth layer. To simplify the problem, only the binary graph is considered here, so according to the threshold Similarity Quantify to generate the edge of the knowledge graph , which represents the relationship between concept nodes in the graph, namely:

[0116]

[0117] Where sgn is a symbolic function, and eventually we will get M layers of prior knowledge graphs of different granularities. , represents the prior model (i.e., the pre-trained video encoder network and the text encoder network ) for downstream datasets The hierarchical prior expression of It is the Mth layer knowledge graph.

[0118] 3. The learning and reasoning module of systematized predefined knowledge is used to sample random walk sequences of fixed walk lengths through deep random path exploration; the obtained random walk sequences are regarded as virtual sentences, and the nodes in the random walk sequences are regarded as virtual words, to learn and reason the node embedding of the graph, and to adopt the idea of ​​missing word prediction context to relax the optimization problem; the attribute information of all nodes of the hierarchical knowledge graph is updated to form the final systematized predefined knowledge graph. Furthermore, the specific operation mode of the learning and reasoning module of systematized predefined knowledge is as follows:

[0119] (1) Through deep random path exploration, a random walk sequence with a fixed walk length is sampled. Furthermore, the step S31 is specifically: using DeepWalk graph embedding technology to embed the m-th layer of knowledge graph The structural relationship in the hierarchical knowledge graph is learned and reasoned; for the mth layer of the knowledge graph , through random path exploration, a random walk sequence with a fixed walk length t is sampled.

[0120] (2) The obtained random walk sequence is regarded as a virtual sentence, and the nodes in the random walk sequence are regarded as virtual words, and the node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted to relax the optimization problem. Furthermore, the S32 is specifically: learning and inferring the m-th layer of knowledge graph Node embedding, that is, learning the following mapping function :

[0121]

[0122] in For nodes, is the mth level knowledge graph The collection of nodes, is the number of nodes, and d is the dimension of node embedding.

[0123] The optimized agent task is: given a sequence of nodes that have been experienced in a random path exploration, predict a certain observation node The maximum likelihood that the next ,Right now:

[0124]

[0125] Represents the mth level knowledge graph The observation nodes in Through the mapping function Mapping is done.

[0126] However, because the random walk sequence has sequential dependencies, as the path exploration length grows, the calculation of this probability becomes very difficult. Therefore, the optimization problem can be relaxed to:

[0127]

[0128] The nodes The subscript indicates the observation node The position of the relevant nodes, the objective function models the length of 2 The transition probability of random path exploration. By optimizing it, the m-th layer knowledge graph The structural information is encoded into the embedding information of each abstract concept node in this layer. .

[0129] (3) Update the attribute information of all nodes in the hierarchical knowledge graph to form the final systematic predefined knowledge graph. The step S33 is specifically as follows: and new systematized embedded information In this embodiment, the hierarchical knowledge graph The attribute information of all nodes is updated to ,in For all original attribute information, Embed all new systematized information to form the final systematized predefined knowledge graph , used to assist downstream small-scale tutorial video datasets The specific tutorial long-term activity analysis task, the specific process is as follows Figure 4 shown.

[0130] 4. Downstream task performance enhancement module, which is used to represent the predefined knowledge and systematize the predefined knowledge graph of downstream query samples of a given modality The similarity calculation is performed on the representation of the corresponding modality of all nodes in each layer, and the concept node with the highest matching degree in each layer of the knowledge graph is found; the representation of the concept node is fused through the self-attention fusion strategy; and then fused with the original representation of the sample of the given modality to obtain the enhanced feature representation of the sample. Furthermore, the specific operation mode of the downstream task performance enhancement module is as follows:

[0131] First, by giving the modality Sample downstream query Predefined knowledge representation With systematic predefined knowledge graph Representation of the corresponding modes of all nodes in each layer Calculate similarity. Considering that each node representation already contains the systematic information of the knowledge graph, we only need to find the concept node with the highest matching degree in each layer of the knowledge graph. ,Right now:

[0132]

[0133] in, Represents the mth level knowledge graph The i-th node in Represents the mth level knowledge graph The i-th node in corresponds to the representation of the mode, It can be various similarity calculation functions, and the simple cosine similarity is used here. Therefore, for the M-layer systematic predefined knowledge graph, a total of M most relevant concept nodes at different levels are matched. , and its corresponding representation These best matches represent Finally, the representations of these M concept nodes are fused through the self-attention fusion strategy. This mechanism can adaptively weight the feature representations of abstract concept nodes at different levels, which has stronger selectivity and adaptability and the best performance. After fusion, it is combined with the original representation of the sample q of the given modality. Fusion, to obtain the enhanced feature representation of sample q.

[0134] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A long-term activity analysis method based on hierarchical predefined knowledge learning, characterized in that: The following steps are involved: S1. Using multimodal semantic consistency, we embed the video clips of tutorial activities and their corresponding text descriptions into a joint semantic space through contrastive learning, and learn the prior distribution of the joint representation of vision and language. S2. Sample some video text pairs from the downstream data set in proportion, cluster their representations, and obtain hierarchical sample clusters from top to bottom to achieve the construction of hierarchical concept nodes; for different granularity levels, construct a corresponding knowledge graph; S3. Through deep random path exploration, random walk sequences with fixed walk lengths are sampled. The obtained random walk sequences are regarded as virtual sentences, and the nodes in the random walk sequences are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted to relax the optimization problem. The attribute information of all nodes in the hierarchical knowledge graph is updated to form the final systematic predefined knowledge graph. S4. Calculate the similarity between the predefined knowledge representation of the downstream query sample of the given modality and the representation of the corresponding modality of all nodes in each layer of the systematic predefined knowledge graph to find the concept node with the highest matching degree in each layer of the knowledge graph; fuse the representation of the concept node through the self-attention fusion strategy; and then fuse it with the original representation of the sample of the given modality to obtain the enhanced feature representation of the sample.

2. The long-term activity analysis method based on hierarchical predefined knowledge learning according to claim 1 is characterized in that: Step S1 is specifically as follows: S11. Using multimodal semantic consistency, we embed the video clips of tutorial activities and their corresponding text descriptions into a d-dimensional joint semantic space through contrastive learning to learn the prior distribution of the joint representation of vision and language. , where f, g are the video encoder network and text encoder network respectively, x, y are the input video clips and the corresponding text descriptions respectively; S12. Construct a contrastive learning agent task and use the softmax version of the noise contrast estimation loss function to self-supervise and learn ; S13, using the MIL-NCE loss function, relax a single positive sample pair into a candidate set of positive sample pairs , obtaining a self-supervised visual-language joint representation prior on the tutorial video dataset HowTo100M.

3. The long-term activity analysis method based on hierarchical predefined knowledge learning according to claim 1 is characterized in that: Step S2 includes the following steps: S21. Constructing hierarchical concept nodes: from downstream datasets In proportion Sample some video text pairs and represent them Hierarchical k-Means algorithm is used for clustering, and hierarchical sample clusters are obtained from top to bottom. They are video representation and text representation respectively; S22. Construct a hierarchical prior knowledge graph: For different levels of granularity m, construct a corresponding knowledge graph ,node By this layer Cluster Centers Composition, among which is the set of nodes of the m-th layer knowledge graph, is the set of edges of the m-th layer knowledge graph, are the cluster centers of the mth layer respectively The video representation and text representation in the , the similarity between any two concept nodes i and j is determined by the joint representation of the video and text of the cluster center. Considering the binary graph, the similarity is quantified according to the threshold to obtain the prior knowledge graphs of different granularities.

4. The long-term activity analysis method based on hierarchical predefined knowledge learning according to claim 3 is characterized in that: Step S21 is specifically as follows: Set the number of clustering levels to , the branch parameters of each layer are , from top to bottom, each layer will get Hierarchical clustering. is the Mth layer branch parameter; each node in each layer is regarded as an abstract concept, representing the overall properties of all samples in the corresponding cluster.

5. The long-term activity analysis method based on hierarchical predefined knowledge learning according to claim 3 is characterized in that: Step S22 is specifically as follows: The similarity between any two concept nodes i and j is determined by the joint representation of the video and text of the cluster center, that is: in is element-wise multiplication, is the video representation in the i-th cluster center of the m-th layer, is the video representation in the jth cluster center of the mth layer, is the text representation in the i-th cluster center of the m-th layer, is the text representation in the jth cluster center of the mth layer; considering the binary graph, according to the threshold Similarity Quantify and generate the edges of the knowledge graph , represents the relationship between concept nodes in the graph, namely: Where sgn is a symbolic function, and M layers of prior knowledge graphs of different granularities are obtained , represents the pre-trained video encoder network and the text encoder network For downstream datasets The hierarchical prior expression of It is the Mth layer knowledge graph.

6. The long-term activity analysis method based on hierarchical predefined knowledge learning according to claim 1 is characterized in that: In step S3, the random walk sequence with a fixed walk length is sampled by deep random path exploration as follows: Use DeepWalk graph embedding technology to embed the m-th layer of knowledge graph The structural relationship in the hierarchical knowledge graph is learned and reasoned; for the mth layer of the knowledge graph , through random path exploration, a random walk sequence with a fixed walk length t is sampled.

7. The long-term activity analysis method based on hierarchical predefined knowledge learning according to claim 6 is characterized in that: In step S3, the obtained random walk sequence is regarded as a virtual sentence, and the nodes in the random walk sequence are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted. The specific relaxation optimization problem is: Learn and reason about the m-th level knowledge graph Node embedding, that is, learning the following mapping function : in For nodes, is the mth level knowledge graph The collection of nodes, is the number of nodes, d is the dimension of node embedding; The optimization agent task is set as follows: given a sequence of nodes that have been experienced in a random path exploration, predict a certain observation node The maximum likelihood that the next ,Right now: Represents the mth level knowledge graph The observation nodes in Through the mapping function Mapping is performed; Using missing words to predict the context, the optimization problem is relaxed to the following objective function: The nodes The subscript indicates the observation node The position of the relevant nodes, the objective function models the length of 2 The transition probability of random path exploration is optimized, and the knowledge graph The structural information is encoded into the embedding information of each abstract concept node in this layer. ; Update the attribute information of all nodes in the hierarchical knowledge graph to form the final systematic predefined knowledge graph: Hierarchical Knowledge Graph The attribute information of all nodes is updated to ,in For all original attribute information, Embed all new systematized information to form the final systematized predefined knowledge graph .

8. A long-term activity analysis system based on hierarchical predefined knowledge learning, characterized in that: Includes the following modules: An unstructured self-supervised representation prior building module that exploits multimodal semantic consistency and embeds video clips of tutorial activities and their corresponding text descriptions into a joint semantic space through contrastive learning to learn a joint visual-linguistic representation prior distribution. A systematic predefined knowledge graph construction module is used to sample some video text pairs from the downstream data set in proportion, cluster their representations, and obtain hierarchical sample clustering clusters from top to bottom to realize the construction of hierarchical concept nodes; for different granularity levels, a corresponding knowledge graph is constructed; The learning and reasoning module of systematic predefined knowledge is used to sample random walk sequences of fixed walk lengths through deep random path exploration. The obtained random walk sequences are regarded as virtual sentences, and the nodes in the random walk sequences are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted to relax the optimization problem. The attribute information of all nodes in the hierarchical knowledge graph is updated to form the final systematic predefined knowledge graph. The downstream task performance enhancement module is used to calculate the similarity between the predefined knowledge representation of the downstream query sample of the given modality and the representation of the corresponding modality of all nodes in each layer of the systematic predefined knowledge graph, and find the concept node with the highest matching degree in each layer of the knowledge graph; fuse the representation of the concept node through the self-attention fusion strategy; and then fuse it with the original representation of the sample of the given modality to obtain the enhanced feature representation of the sample.

9. The long-term activity analysis system based on hierarchical predefined knowledge learning according to claim 8, characterized in that: The unsystematic self-supervised representation prior construction module is used to utilize multimodal semantic consistency to embed the video clips of the tutorial activities and their corresponding text descriptions into a joint semantic space through contrastive learning, and learn the visual language joint representation prior distribution as follows: Using multimodal semantic consistency, we embed the video clips of tutorial activities and their corresponding text descriptions into a d-dimensional joint semantic space through contrastive learning to learn the prior distribution of the joint visual-linguistic representation. , where f, g are the video encoder network and text encoder network respectively, x, y are the input video clips and the corresponding text descriptions respectively; Construct a contrastive learning agent task and use the softmax version of the noise contrast estimation loss function to self-supervise and learn ; Using the MIL-NCE loss function, a single positive sample pair is relaxed into a candidate set of positive sample pairs. , obtaining a self-supervised visual-language joint representation prior on the tutorial video dataset HowTo100M.

10. The long-term activity analysis system based on hierarchical predefined knowledge learning according to claim 8, characterized in that: In the learning and reasoning module of the systematic predefined knowledge, a random walk sequence with a fixed walk length is sampled by deep random path exploration as follows: Use DeepWalk graph embedding technology to embed the m-th layer of knowledge graph The structural relationship in the hierarchical knowledge graph is learned and reasoned; for the mth layer of the knowledge graph , through random path exploration, a random walk sequence with a fixed walk length t is sampled; The obtained random walk sequence is regarded as a virtual sentence, and the nodes in the random walk sequence are regarded as virtual words. The node embedding of the graph is learned and inferred, and the idea of ​​missing word prediction context is adopted. The specific relaxation optimization problem is: Learn and reason about the m-th level knowledge graph Node embedding, that is, learning the following mapping function : in For nodes, is the mth level knowledge graph The collection of nodes, is the number of nodes, d is the dimension of node embedding; The optimization agent task is set as follows: given a sequence of nodes that have been experienced in a random path exploration, predict a certain observation node The maximum likelihood that the next ,Right now: Represents the mth level knowledge graph The observation nodes in Through the mapping function Mapping is performed; Using missing words to predict the context, the optimization problem is relaxed to the following objective function: The nodes The subscript indicates the observation node The position of the relevant nodes, the objective function models the length of 2 The transition probability of random path exploration is optimized, and the m-th layer knowledge graph The structural information is encoded into the embedding information of each abstract concept node in this layer. ; Update the attribute information of all nodes in the hierarchical knowledge graph to form the final systematic predefined knowledge graph: Hierarchical Knowledge Graph The attribute information of all nodes is updated to ,in For all original attribute information, Embed all new systematized information to form the final systematized predefined knowledge graph .

Images