Weakly supervised temporal action localization method and device with external knowledge features

Abstract

The application provides a weakly supervised time action localization method and device introducing external knowledge features, and the method comprises the following steps: cutting a video to be recognized to obtain a plurality of video clips, inputting the video clips into a feature extraction module of an action localization model to extract features of the video clips, and obtaining clip features; inputting the clip features into an attention module of the action localization model to obtain initial attention features of the clip features; performing object recognition on key frames in the video clips to obtain object recognition results in the key frames, obtaining external knowledge features based on the object recognition results, and the external knowledge features being knowledge based on the action localization model; obtaining target attention features based on the external knowledge features and the initial attention features, inputting the target attention features into a classification module in the action localization model, and obtaining action localization results of the video to be recognized output by the classification module. The application can improve the accuracy of weakly supervised time action localization.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a weakly supervised temporal action localization method and device that introduces external knowledge features. Background Art

[0002] Temporal action localization is one of the important tasks in video understanding. The goal is to locate the temporal boundaries of actions in videos and simultaneously identify the action categories. In recent years, thanks to the development of deep models, supervised temporal action localization has achieved good performance, which is obviously highly dependent on the quantity and quality of manually annotated samples. To this end, localization has emerged in the weakly supervised setting, where only video-level labels are available. Weakly-supervised Temporal Action Localization (WS-TAL) is a partially supervised task that aims to localize action instances supervised by video-level action labels.

[0003] Existing techniques for weakly supervised temporal action localization suffer from low accuracy. This is essentially due to the difficulty of ensuring consistency between instance-level data and video-level labels using coarse-grained label information. This problem stems primarily from a lack of precise annotation information, which limits task performance. Summary of the Invention

[0004] The present invention provides a method and device for weakly supervised temporal action localization by introducing external knowledge features, so as to solve the defect of low accuracy of positioning results of weakly supervised temporal action localization in the prior art and improve the accuracy of positioning results of weakly supervised temporal action localization.

[0005] The present invention provides a weakly supervised temporal action localization method that introduces external knowledge features, comprising:

[0006] Obtain a video to be identified, segment the video to be identified to obtain multiple video segments, and input the video segments into a feature extraction module of an action localization model to extract features from the video segments to obtain segment features;

[0007] Inputting the segment features into the attention module of the action localization model to obtain initial attention features;

[0008] Performing object recognition on key frames in the video clip to obtain object recognition results in the key frames, and obtaining external knowledge features based on the object recognition results, where the external knowledge features are based on knowledge outside the action localization model;

[0009] Based on the external knowledge features and the initial attention features, a target attention feature is obtained, and the target attention feature is input into the classification module in the action localization model to obtain the action localization result of the video to be identified output by the classification module.

[0010] According to a weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention, the external knowledge features include a first knowledge feature; the external knowledge features obtained based on the object recognition result include:

[0011] Determining the action category relationship value corresponding to the key frame based on the pre-established relationship values ​​between each object and each preset action category and each object in the object recognition result of the key frame;

[0012] The first knowledge feature is obtained based on the action category relationship value corresponding to each of the key frames.

[0013] According to a weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention, before obtaining the first knowledge feature based on the action category relationship value corresponding to each key frame, the method includes:

[0014] When the object recognition result of the key frame only includes the target object, setting the action category relationship value corresponding to the key frame to 0;

[0015] The target object is an object associated with all of the preset action categories.

[0016] According to a weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention, the external knowledge features include a second knowledge feature; and the external knowledge features obtained based on the object recognition result include:

[0017] Generate a question text based on the object recognition result, the key frame, and each preset action category;

[0018] The question text is input into a question-answering model, a probability value of each of the preset action categories included in the key frame output by the question-answering model is obtained, and the second knowledge feature is obtained based on the probability value.

[0019] According to a weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention, obtaining a target attention feature based on the external knowledge feature and the initial attention feature includes:

[0020] The external knowledge feature and the initial attention feature are weightedly fused to obtain the target attention feature.

[0021] According to a weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention, the target attention feature is obtained based on the external knowledge feature and the initial attention feature, including:

[0022] When the external knowledge feature corresponds to an action of a preset action category that does not exist in the key frame, the external knowledge feature is set to 0.

[0023] The present invention also provides a weakly supervised temporal action localization device that introduces external knowledge features, comprising:

[0024] An initial feature module is used to obtain a video to be identified, segment the video to be identified to obtain multiple video segments, and input the video segments into a feature extraction module of an action localization model to extract features from the video segments to obtain segment features;

[0025] An attention initial feature module, configured to input the segment features into the attention module of the action localization model to obtain initial attention features;

[0026] an external knowledge module, configured to perform object recognition on key frames in the video clip to obtain object recognition results in the key frames, and obtain external knowledge features based on the object recognition results, wherein the external knowledge features are based on knowledge outside the action localization model;

[0027] A positioning result output module is used to obtain a target attention feature based on the external knowledge feature and the initial attention feature, input the target attention feature into the classification module in the action positioning model, and obtain the action positioning result of the video to be identified output by the classification module.

[0028] The present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, any one of the above-mentioned weakly supervised temporal action localization methods that introduces external knowledge features is implemented.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned weakly supervised temporal action localization methods that introduce external knowledge features.

[0030] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned weakly supervised temporal action localization methods that introduce external knowledge features.

[0031] The present invention provides a weakly supervised temporal action localization method and device that introduces external knowledge features. In the process of temporal action localization, in addition to using the information mined by the action localization model, object recognition is also performed on key frames in the video segment. Based on the results of object recognition, knowledge outside the action localization model is obtained, thereby realizing the transfer of external knowledge to weakly supervised temporal action localization and improving the accuracy of weakly supervised temporal action localization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 This is a flow chart of the weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention.

[0034] Figure 2 This is a schematic diagram of the data processing flow in the weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention.

[0035] Figure 3 This is a schematic diagram of the first knowledge feature generation process in the weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention.

[0036] Figure 4 This is a schematic diagram of the second knowledge feature generation process in the weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention.

[0037] Figure 5 This is an example diagram of the effect of the weakly supervised temporal action localization method that introduces external knowledge features provided by the present invention.

[0038] Figure 6 It is a structural diagram of the weakly supervised temporal action localization device that introduces external knowledge features provided by the present invention.

[0039] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0041] The following combination Figure 1-Figure 5 The weakly supervised temporal action localization method provided by the present invention that introduces external knowledge features is described as follows: Figure 1 As shown, the method includes the steps of:

[0042] S110, obtaining a video to be identified, segmenting the video to be identified to obtain multiple video segments, and inputting the video segments into a feature extraction module of an action localization model to extract features from the video segments to obtain segment features;

[0043] S120, inputting the segment features into the attention module of the action localization model to obtain initial attention features of each segment feature;

[0044] S130, performing object recognition on key frames in the video clip to obtain object recognition results in the key frames, and obtaining external knowledge features based on the object recognition results, where the external knowledge features are knowledge other than the action localization model;

[0045] S140. Based on the external knowledge features and the initial attention features, a target attention feature is obtained, and the target attention feature is input into a classification module in the action localization model to obtain an action localization result of the video to be identified output by the classification module.

[0046] The action localization model in the method provided by the present invention is a weakly supervised temporal action localization model, that is, the training process of the action localization model in the method provided by the present invention is carried out in a weakly supervised manner, and the labels in the training data include video-level action categories. When it is necessary to perform temporal action localization on the video to be identified, the video to be identified is first segmented to obtain multiple video segments, each of which does not overlap with each other, and each video segment is input into the action localization model for temporal action localization. The action localization model includes a feature extraction module, an attention module, and a classification module. The feature extraction module is used to extract features of each video segment, generally including RGB features and optical flow features. The feature extraction module can use existing feature extraction networks, such as C3D, I3D, TSN, UntrimmedNet, etc. Figure 2As shown in the figure, after obtaining the segment features of each video clip, the segment features are input into the attention module to obtain the initial attention features. The classification module is used to classify the video clips into actions, that is, to determine which action the video clip corresponds to, thereby achieving action localization of the video to be identified. In other words, the action localization results of the video to be identified output by the classification module include the action category of each video clip.

[0047] In the method provided by the present invention, after obtaining the initial attention features, classification is not performed directly based on the initial attention features. Instead, external knowledge features are introduced. Based on the external knowledge features and the initial attention features, target attention features are obtained. The target attention features are input into the classification module to obtain the action localization results. In this way, when performing the temporal action localization task, not only the knowledge mined from the training data by the action localization model is utilized, but also knowledge external to the action localization model is utilized, achieving the transfer of external knowledge to weakly supervised temporal action localization and improving the accuracy of weakly supervised temporal action localization.

[0048] like Figure 2 As shown, the external knowledge features in the method provided by the present invention are obtained by performing object recognition on key frames in video clips. Object recognition can be achieved by using existing trained object detectors (e.g. Figure 2 This is achieved using the Faster RCNN in

[15] . External knowledge features are then obtained based on the object recognition results of key frames.

[0049] Specifically, the external knowledge features include a first knowledge feature, which is obtained based on the object recognition result and includes:

[0050] Determine the action category relationship value corresponding to the key frame based on the pre-established relationship value between each object and each preset action category and each object in the object recognition result of the key frame;

[0051] A first knowledge feature is obtained based on the action category relationship value corresponding to each key frame.

[0052] The external knowledge features in the method provided by the present invention may include a first knowledge feature, which is derived from explicit knowledge. Explicit knowledge refers to clear, complete truth that can be expressed using a symbolic system, including concepts, viewpoints, principles, norms, processes, key points, and the like. The present invention uses knowledge graphs as explicit knowledge. Knowledge graphs include Wikipedia, WordNet, ConceptNet, and the like. Knowledge graphs have rich semantic information and highly integrated information, which can be quickly introduced into different tasks in a certain manner, helping the model significantly improve performance.

[0053] Most actions involve interactions with objects. For example, when playing tennis, there might be a ball, a racket, and a net. This interactive relationship is crucial for action recognition. However, existing weakly supervised temporal action localization methods typically only consider global information, while ignoring the importance of local object regions. This paper uses knowledge graphs to introduce the relationship between actions and related objects as explicit knowledge.

[0054] Specifically, the method provided by this invention utilizes a knowledge graph to construct relationship values ​​between each object and various preset action categories. These preset action categories are pre-defined based on the application scenario of the method provided by this invention. For example, in a sports scenario, seven preset action categories can be set: baseball pitching, basketball slam dunking, frisbee catching, golf swinging, soccer free throws, tennis swinging, and volleyball spiking. The action localization model then outputs which of these preset action categories the video clip corresponds to.

[0055] According to the preset action categories, based on the knowledge graph, the objects involved in the preset action categories can be pre-constructed. For example, the baseball pitching action category may involve people, baseball bats, and baseball gloves. The relationship value between the preset action categories and the involved objects is also constructed, such as Figure 3 As shown, the baseball pitching action category has a relationship value of 0.01 with the person, 0.041 with the baseball bat, and 0.046 with the baseball glove. For an object detected in a keyframe, an action category relationship value reflecting the likelihood of the preset action category occurring in the keyframe is obtained based on the relationship value between the object and the preset action category. The calculation formula for the action category relationship value of a keyframe can be expressed as:

[0056] ;

[0057] in, represents the first knowledge feature, represents the relationship value between the preset action category i and the object j, M represents the set of all objects detected in the key frame, and S represents the set of objects involved in all preset action categories.

[0058] The action category relationship values ​​of all key frames are combined into a vector to obtain the first external knowledge feature.

[0059] Furthermore, in the same scene, the preset action categories may all be associated with the same object. For example, athletes are often seen in sports activities. Therefore, when there is only a person in the picture, it is impossible to confirm whether the preset action has occurred. To avoid the impact of this situation on the action positioning result, in one implementation of the method provided by the present invention, before obtaining the first knowledge feature based on the action category relationship value corresponding to each key frame, the method includes:

[0060] When the object recognition result of a key frame only includes the target object, the action category relationship value corresponding to the key frame is set to 0;

[0061] The target object is an object associated with all preset action categories.

[0062] Assuming that the target object is a person and the relationship value between the target object and each preset action category is 0.01, the above process can be expressed as follows:

[0063] .

[0064] On the other hand, implicit knowledge can also be used as external knowledge, that is, the external knowledge feature includes a second knowledge feature, and the second knowledge feature is implicit knowledge. Implicit knowledge refers to knowledge that is difficult to describe using a symbolic system. It originates from personal experience and is closely related to personal beliefs, opinions, and values. In machine learning, implicit external knowledge can be extended to knowledge that cannot be queried, is difficult to provide examples for individual knowledge, and mainly comes from model learning outside the main task. In existing temporal action localization methods, the model only uses visual information to identify actions. However, in fact, video tags (preset action categories) contain a lot of useful information. The method provided by the present invention is to mine implicit knowledge in videos. Specifically, external knowledge features are obtained based on object recognition results, including:

[0065] Generate question text based on object recognition results, key frames, and various preset action categories;

[0066] The question text is input into the question-answering model, and the probability values ​​of each preset action category included in the key frame output by the question-answering model are obtained, and the second knowledge feature is obtained based on the probability value.

[0067] Generating question text can be done based on templates, such as Figure 4 As shown, the object recognition results, keyframes, and various preset action categories are entered into the template to generate the question text. For example, this picture shows a tennis ball and a person. Which sport does this picture represent: baseball pitching, basketball slam dunking, catching a Frisbee, golf swinging, soccer free throw shooting, tennis swinging, or volleyball spiking?

[0068] The question-answering model can be an existing question-answering model, such as VL-BERT fine-tuned on the VQA v2.0 and ImageNet datasets. Based on the probability values ​​of each preset action category included in the keyframes output by the question-answering model, the probability values ​​of each preset action category included in the keyframes are formed into a vector, and the vectors corresponding to all keyframes are combined to obtain the second knowledge feature.

[0069] After obtaining the external knowledge features, the target attention features are obtained based on the external knowledge features and the initial attention features, specifically including:

[0070] The external knowledge features and the initial attention features are weightedly fused to obtain the target attention features.

[0071] When the external knowledge features only include the first knowledge features or the second knowledge features, they can be merged into the initial attention features in a weighted manner, as shown in the following formula:

[0072] ;

[0073] or, ;

[0074] in, represents the initial attention features corresponding to the video clip, 、 represent the target attention features of the video clip obtained based on the first knowledge feature and the second knowledge feature, respectively. A vector of probability values ​​representing each preset action category in the key frames of the video clip, 、 、 、 is the weight of the feature.

[0075] When the external knowledge feature includes the first knowledge feature and the second knowledge feature, the formula for obtaining the target attention feature can be expressed as:

[0076] ;

[0077] in, represents the target attention feature of the video clip obtained based on the first knowledge feature and the second knowledge feature, 、 is the weight of the feature.

[0078] In a possible implementation, when the external knowledge features include the first knowledge feature and the second knowledge feature, two methods can be used for fusion. The first method is the union ∪ method, which directly adds the weights to the two external knowledge features, that is, , 、 Represents the weight, which can ensure the integrity of the information and avoid errors in the final result caused by errors in a certain type of knowledge. The other is the intersection ∩ method, that is, based on the external knowledge features and the initial attention features, the target attention features are obtained, including:

[0079] When there is no action of the preset action category in the key frame corresponding to the external knowledge feature, the external knowledge feature is set to 0.

[0080] That is, if and only if two pieces of external knowledge simultaneously determine that there is a behavior in the key frame, the two are weighted and summed, otherwise it is set to 0. This process can be expressed as:

[0081] .

[0082] The absence of a preset action category in the keyframe corresponding to the external knowledge feature means that when the first external knowledge feature is acquired, there is no object associated with the preset action category in the keyframe, or only the target object is present, and when the second external knowledge feature is acquired, the probability values ​​of each preset action category in the keyframe output by the question-answering model are all 0 or below a preset threshold. This avoids the introduction of noise information unrelated to the action.

[0083] The method provided by the present invention was experimentally verified, and the experimental setup was as follows:

[0084] Data: Experiments were conducted on the THUMOS'14 dataset, a commonly used benchmark for WS-TAL. Its videos are from the UCF101 dataset. It includes 20 untrimmed videos from 430 motion categories. Following previous work, we used 200 validation videos as the training set and 213 test videos for testing.

[0085] Evaluation Metrics: The evaluation metric used in the experiments is mAP (mean average precision), with different IoU (intersection over union) thresholds ranging from 0.1 to 0.9, which is commonly used in WS-TAL. The official evaluation code is used for evaluation.

[0086] Implementation details: Two external knowledge features are used, and most experimental settings are consistent with existing WS-TAL methods. All experiments are performed on NVIDIA GTX1080Ti and Tesla V100-SXM2-32GB.

[0087] The method provided by the present invention is compared with the most advanced WS-TAL method on THUMOS'14. The results are shown in Table 1.

[0088] Table 1

[0089]

[0090] Since we used CO2-Net and TFE-DCN as baseline methods, we first reproduced their results based on their code and experimental settings, which differ slightly from the performance reported in the corresponding papers. These are marked with * and used as baselines for experimental comparison. As can be seen in Table 1, our results improve by 1.2 percentage points over CO2-Net and by 0.5 percentage points over TFE-DCN. These results demonstrate the effectiveness of our method, which achieves competitive performance compared to other methods.

[0091] As mentioned earlier, based on existing datasets, we can extract knowledge related to seven actions in the THUMOS'14 dataset. These actions are: baseball pitching, basketball slam dunking, Frisbee catching, golf swinging, soccer free throws, tennis swinging, and volleyball spiking. For now, we will consider using knowledge from only these seven data types as a validation approach.

[0092] First, we used the CO2-Net method to conduct ablation experiments on these seven types of data. The results are shown in Table 2. "+KG" represents the introduction of knowledge from the knowledge graph, "+VLP" represents the introduction of knowledge from the vision-language pre-trained model, and "∩" and "∪" represent two methods of knowledge fusion. Table 2 shows that both types of knowledge can improve performance to varying degrees, and using a combined knowledge fusion approach is more conducive to ensuring model performance.

[0093] Table 2

[0094]

[0095] Ablation experiments were also conducted on all THUMOS'14 datasets, but only seven types of knowledge were added. Table 3 shows the ablation experiments for the two types of knowledge. It can be seen that introducing a single type of knowledge improves performance, but the improvement is not significant. Using both types of knowledge simultaneously improves performance, and the union fusion method is more effective than the intersection method. This is because using the intersection method results in the mutual suppression of the two types of knowledge and the loss of more useful information.

[0096] Table 3

[0097]

[0098] Finally, the comparison results of the confidence scores based on the two external knowledge and the true values ​​are as follows Figure 5 As shown, Figure 5 The light-colored line and the dark-colored line represent the results of external knowledge and true values, respectively. The horizontal axis represents the keyframes, and the vertical axis represents the probability of the operation occurring. Figure 5The left side of the figure shows the result of linear external knowledge, and the right side shows the result of implicit external knowledge. Due to the difference in the scores of the two types of knowledge, in order to facilitate the comparison of the results, the true values ​​are set to 0.5 and 2 respectively. Figure 5 It can be seen that the confidence score based on external knowledge is highly consistent with the true value of the annotation, indicating that the external knowledge in the method provided by the present invention is of great value to WS-TAL.

[0099] The following describes the weakly supervised temporal action localization device that introduces external knowledge features provided by the present invention. The weakly supervised temporal action localization device that introduces external knowledge features described below and the weakly supervised temporal action localization method that introduces external knowledge features described above can refer to each other. Figure 6 As shown, the weakly supervised temporal action localization device with external knowledge features provided by the present invention includes:

[0100] The initial feature module 610 is used to obtain a video to be identified, segment the video to be identified to obtain multiple video segments, and input the video segments into the feature extraction module of the action localization model to extract features from the video segments and obtain segment features;

[0101] An attention initial feature module 620 is used to input the segment features into the attention module of the action localization model to obtain initial attention features;

[0102] The external knowledge module 630 is configured to perform object recognition on key frames in the video clip, obtain object recognition results in the key frames, and obtain external knowledge features based on the object recognition results. The external knowledge features are knowledge based on knowledge other than the action localization model.

[0103] The positioning result output module 640 is used to obtain the target attention feature based on the external knowledge feature and the initial attention feature, input the target attention feature into the classification module in the action positioning model, and obtain the action positioning result of the video to be identified output by the classification module.

[0104] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a weakly supervised temporal action localization method that introduces external knowledge features. The method includes: obtaining a video to be identified, segmenting the video to be identified to obtain multiple video segments, inputting the video segments into a feature extraction module of an action localization model to extract features from the video segments and obtain segment features; inputting the segment features into an attention module of the action localization model to obtain initial attention features; performing object recognition on key frames in the video segments to obtain object recognition results in the key frames, obtaining external knowledge features based on the object recognition results, and the external knowledge features are based on knowledge outside the action localization model; obtaining target attention features based on the external knowledge features and the initial attention features, inputting the target attention features into a classification module in the action localization model, and obtaining an action localization result of the video to be identified output by the classification module.

[0105] Furthermore, the logic instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the weakly supervised temporal action localization method that introduces external knowledge features provided by the above methods. The method includes: obtaining a video to be identified, dividing the video to be identified to obtain multiple video segments, inputting the video segments into a feature extraction module of an action localization model to extract features from the video segments and obtain segment features; inputting the segment features into an attention module of an action localization model to obtain initial attention features; performing object recognition on key frames in the video segments to obtain object recognition results in the key frames, and obtaining external knowledge features based on the object recognition results, which are knowledge based on knowledge outside the action localization model; obtaining target attention features based on the external knowledge features and the initial attention features, and inputting the target attention features into a classification module in the action localization model to obtain the action localization results of the video to be identified output by the classification module.

[0107] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the weakly supervised temporal action localization method that introduces external knowledge features provided by the above-mentioned methods, the method comprising: obtaining a video to be identified, dividing the video to be identified to obtain multiple video segments, inputting the video segments into a feature extraction module of an action localization model to extract features from the video segments and obtain segment features; inputting the segment features into an attention module of an action localization model to obtain initial attention features; performing object recognition on key frames in the video segments to obtain object recognition results in the key frames, and obtaining external knowledge features based on the object recognition results, the external knowledge features being knowledge based outside the action localization model; obtaining target attention features based on the external knowledge features and the initial attention features, and inputting the target attention features into a classification module in the action localization model to obtain the action localization results of the video to be identified output by the classification module.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0109] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A weakly supervised temporal action localization method that introduces external knowledge features, characterized by: include: Obtain a video to be identified, segment the video to be identified to obtain multiple video segments, and input the video segments into a feature extraction module of an action localization model to extract features from the video segments to obtain segment features; Inputting the segment features into the attention module of the action localization model to obtain initial attention features; Performing object recognition on key frames in the video clip to obtain object recognition results in the key frames, and obtaining external knowledge features based on the object recognition results, where the external knowledge features are based on knowledge outside the action localization model; Based on the external knowledge feature and the initial attention feature, a target attention feature is obtained, the target attention feature is input into a classification module in the action localization model, and an action localization result of the video to be identified is obtained from the classification module; The external knowledge feature includes a first knowledge feature; The obtaining of external knowledge features based on the object recognition result includes: Determining the action category relationship value corresponding to the key frame based on the pre-established relationship values ​​between each object and each preset action category and each object in the object recognition result of the key frame; Obtaining the first knowledge feature based on the action category relationship value corresponding to each of the key frames; The external knowledge feature includes a second knowledge feature; the external knowledge feature obtained based on the object recognition result includes: Generate a question text based on the object recognition result, the key frame, and each preset action category; Inputting the question text into a question-answering model, obtaining a probability value of each of the preset action categories in the key frame output by the question-answering model, and obtaining the second knowledge feature based on the probability value; The obtaining of a target attention feature based on the external knowledge feature and the initial attention feature includes: The external knowledge feature and the initial attention feature are weightedly fused to obtain the target attention feature.

2. The weakly supervised temporal action localization method using external knowledge features according to claim 1 is characterized in that: Before obtaining the first knowledge feature based on the action category relationship value corresponding to each of the key frames, the method includes: When the object recognition result of the key frame only includes the target object, setting the action category relationship value corresponding to the key frame to 0; The target object is an object associated with all of the preset action categories.

3. The weakly supervised temporal action localization method using external knowledge features according to claim 1 is characterized in that: The obtaining of a target attention feature based on the external knowledge feature and the initial attention feature includes: When the external knowledge feature corresponds to an action of a preset action category that does not exist in the key frame, the external knowledge feature is set to 0.

4. A weakly supervised temporal action localization device that introduces external knowledge features, characterized in that: The device comprises: An initial feature module is used to obtain a video to be identified, segment the video to be identified to obtain multiple video segments, and input the video segments into a feature extraction module of an action localization model to extract features from the video segments to obtain segment features; An attention initial feature module, configured to input the segment features into the attention module of the action localization model to obtain initial attention features; an external knowledge module, configured to perform object recognition on key frames in the video clip to obtain object recognition results in the key frames, and obtain external knowledge features based on the object recognition results, wherein the external knowledge features are based on knowledge outside the action localization model; A positioning result output module is used to obtain a target attention feature based on the external knowledge feature and the initial attention feature, input the target attention feature into the classification module in the action positioning model, and obtain the action positioning result of the video to be identified output by the classification module; The external knowledge feature includes a first knowledge feature; the external knowledge feature obtained based on the object recognition result includes: Determining the action category relationship value corresponding to the key frame based on the pre-established relationship values ​​between each object and each preset action category and each object in the object recognition result of the key frame; Obtaining the first knowledge feature based on the action category relationship value corresponding to each of the key frames; The external knowledge feature includes a second knowledge feature; the external knowledge feature obtained based on the object recognition result includes: Generate a question text based on the object recognition result, the key frame, and each preset action category; Inputting the question text into a question-answering model, obtaining a probability value of each of the preset action categories in the key frame output by the question-answering model, and obtaining the second knowledge feature based on the probability value; The obtaining of a target attention feature based on the external knowledge feature and the initial attention feature includes: The external knowledge feature and the initial attention feature are weightedly fused to obtain the target attention feature.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for weakly supervised temporal action localization by introducing external knowledge features as described in any one of claims 1 to 3 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for weakly supervised temporal action localization by introducing external knowledge features as described in any one of claims 1 to 3 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for weakly supervised temporal action localization by introducing external knowledge features as described in any one of claims 1 to 3 is implemented.

Images