Video recognition method, device and storage medium

Abstract

Embodiments of the present application provide a video recognition method, device and storage medium, the method comprising: a recognition device obtaining high-level features of a to-be-recognized video and video frame features of each video frame in the video. The high-level features are used to reflect the overall semantics of the to-be-recognized video, and the video frame features of the multiple video frames respectively reflect the context relationship of different frames of the to-be-recognized video in time sequence. Then, semantic prompt information and time sequence prompt information corresponding to the to-be-recognized video are determined, and finally the recognition of the to-be-recognized video is realized according to the two kinds of prompt information and the high-level features. The semantic prompt information and the time sequence prompt information can be used as a prompt to guide the direction of the recognition device using the high-level features for video recognition. In the above process, the use of the two kinds of prompt information of semantics and context information can provide more accurate and rich guidance direction for the recognition device, so that the recognition device can more accurately perform video recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a video recognition method, device, and storage medium. Background Technology

[0002] Artificial intelligence technology has enabled video recognition, which involves identifying the posture of objects within a video. Video recognition can be applied in numerous scenarios, such as extended reality (XR) scenarios, including virtual reality (VR), augmented reality (AR), and mixed reality (MR), where recognizing user actions in the video enables haptic interaction. Another example is in security scenarios, where identifying suspicious individuals in security videos captured by cameras helps determine their presence.

[0003] Therefore, in the practical application of video recognition, how to improve the accuracy of video recognition has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a video recognition method, device, and storage medium to ensure the accuracy of video recognition.

[0005] In a first aspect, embodiments of the present invention provide a video recognition method, comprising:

[0006] Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified;

[0007] Based on the video frame features, determine the timing prompt information corresponding to the video to be identified;

[0008] The recognition result of the video to be recognized is determined based on the semantic prompt information corresponding to the video to be recognized, the temporal prompt information, and the high-level features.

[0009] Secondly, embodiments of the present invention provide a video recognition method, including:

[0010] In response to input commands on the user interface, acquire the video to be recognized;

[0011] Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified;

[0012] Based on the video frame features, determine the timing prompt information corresponding to the video to be identified;

[0013] The recognition result of the video to be recognized is determined based on the semantic prompt information corresponding to the video to be recognized, the temporal prompt information, and the high-level features.

[0014] The recognition results are displayed on the user interface.

[0015] Thirdly, embodiments of the present invention provide a video recognition method, applied to augmented reality (AR) devices or virtual reality (VR) devices, including:

[0016] Collect videos containing user actions to be identified;

[0017] Obtain high-level features of the video to be identified and video frame features of the video frames in the video to be identified;

[0018] Based on the video frame features, determine the timing prompt information corresponding to the video to be identified;

[0019] Based on the semantic prompt information corresponding to the video to be identified, the temporal prompt information, and the high-level features, the action information contained in the video to be identified is determined;

[0020] Display the image corresponding to the action information.

[0021] Fourthly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the video recognition method of any one of the first to third aspects described above. The electronic device may also include a communication interface for communicating with other devices or communication systems.

[0022] Fifthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, wherein when the executable code is executed by a processor of an electronic device, the processor is able to implement at least any one of the video recognition methods described in the first to third aspects above.

[0023] In the video recognition method provided by this invention, the recognition device acquires the high-level features of the video to be recognized and the video frame features of each video frame in the video. The high-level features reflect the overall semantics of the video to be recognized, while the individual video frame features reflect the temporal correlation between different video frames, i.e., they reflect the contextual information between different video frames. Then, the recognition device determines the semantic and temporal cue information corresponding to the video to be recognized, and finally achieves the recognition of the video to be recognized based on the two types of cue information and the high-level features.

[0024] In the above process, semantic and temporal cues are used together to guide the recognition device in its direction of video recognition. Compared to images, video is a dynamic image; therefore, utilizing both semantic and contextual cues can provide the recognition device with more accurate and richer guidance, enabling it to perform video recognition more accurately. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a video recognition method provided in an embodiment of the present invention;

[0027] Figure 2 A flowchart illustrating a method for determining semantic prompt information provided in an embodiment of the present invention;

[0028] Figure 3 A schematic diagram illustrating a video recognition method provided in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of an information pool training process provided in an embodiment of the present invention;

[0030] Figure 5 A flowchart of another video recognition method provided in an embodiment of the present invention;

[0031] Figure 6 A schematic diagram illustrating another video recognition method provided in an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of another information pool training process provided in an embodiment of the present invention;

[0033] Figure 8 A flowchart illustrating yet another video recognition method provided in an embodiment of the present invention;

[0034] Figure 9 This is a schematic diagram of a video recognition operation interface provided in an embodiment of the present invention;

[0035] Figure 10 A flowchart illustrating yet another video recognition method provided in an embodiment of the present invention;

[0036] Figure 11 This is a schematic diagram illustrating the use of the video recognition method provided in this embodiment of the invention in a VR scenario;

[0037] Figure 12 A schematic diagram illustrating the use of the video recognition method provided in this embodiment of the invention in a security scenario;

[0038] Figure 13 This is a schematic diagram of the structure of a video recognition device provided in an embodiment of the present invention;

[0039] Figure 14 This is a schematic diagram of another video recognition device provided in an embodiment of the present invention;

[0040] Figure 15 This is a schematic diagram of the structure of another video recognition device provided in an embodiment of the present invention;

[0041] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0042] Figure 17 This is a schematic diagram of the structure of another electronic device provided in an embodiment of the present invention;

[0043] Figure 18 This is a schematic diagram of another electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0046] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0047] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to identification.” Similarly, depending on the context, the phrases “if determination” or “if identification (of the condition or event of the statement)” can be interpreted as “when determination” or “in response to determination” or “when identification (of the condition or event of the statement)” or “in response to identification (of the condition or event of the statement).”

[0048] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0049] Before describing the video recognition method provided in the following embodiments of the present invention in detail, the relevant concepts involved in the following embodiments can also be explained:

[0050] A pre-trained model is a model trained on and stored using a large amount of data. By executing historical tasks, this model acquires some historical functionality. When new functionality is introduced, instead of training a new model from scratch, the model can be directly built upon this pre-trained model to perform incremental tasks corresponding to the new functionality, thus enabling the model to acquire the new functionality.

[0051] The prompt information guides the pre-trained model in its recognition direction based on the original input data. The prompt information, along with the original input data, can be fed into the pre-trained model to form the latest input data, allowing the model to recognize the latest input data. The prompt information guides the pre-trained model's recognition direction.

[0052] Video recognition: Video recognition can be the identification of actions of target objects in a video. Optionally, pre-trained models can be used for video recognition.

[0053] Incremental learning from videos: New actions are constantly being generated, so the new actions can be used as new tasks for training. This means that a video sample set containing the new actions needs to be obtained and trained based on this video sample set so that the model can correctly recognize the new actions.

[0054] Based on the above description, some embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0055] Figure 1 This is a flowchart illustrating a video recognition method provided in an embodiment of the present invention. The video recognition method provided in this embodiment can be executed by a recognition device with data processing capabilities. Figure 1 As shown, the method may include the following steps:

[0056] S101, Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified.

[0057] The recognition device first acquires the video to be recognized and then uses its configured encoder to encode the video, thereby obtaining the high-level features of the video and the individual video frame features of different video frames within the video. The high-level features reflect the overall semantics of the video. The individual video frame features reflect the temporal relationships between different video frames in the video, i.e., the contextual relationships between different video frames.

[0058] For acquiring video frame features, optionally, the video encoder in the recognition device can encode each frame in the video to be recognized, and the encoding result can be directly used as the video frame feature. However, considering the computational load of the encoding process and the subsequent computational load of the video recognition process, optionally, the recognition device can also first sample the video to be recognized, and then the encoder can encode the sampled few video frames to obtain the video frame feature. Optionally, the sampling method can be uniform sampling or any other feasible sampling method, and this invention is not limited thereto. Optionally, the number of video frames obtained through sampling can be 8 frames, 16 frames, 24 frames, etc.

[0059] S102, Based on the characteristics of the video frames, determine the timing prompt information corresponding to the video to be identified.

[0060] Next, the recognition device can determine the semantic prompts corresponding to the video, and can also determine the temporal prompts corresponding to the video based on the video frame features obtained from the above encoding. The semantic prompts guide the recognition device to focus on the semantics of the video during recognition. Similarly, the temporal prompts guide the recognition device to focus on the contextual relationships between different video frames during video recognition. The video frame features, high-level features, and both types of prompts can all be represented in the form of feature vectors.

[0061] For determining semantic prompts, one optional approach is for the recognition device to filter out the corresponding semantic prompts from an information pool based on the high-level features of the video to be recognized. Optionally, the information pool can store pairs of information indices and semantic prompts; that is, an information index points to a semantic prompt, and an information index and its pointed-to semantic prompt can be represented as key-value pairs. Furthermore, compared to other semantic prompts in the information pool, the selected semantic prompts should have the highest similarity between their corresponding information indices and the high-level features of the video to be recognized.

[0062] Similarly, for determining timing prompt information, one optional method is for the recognition device to filter out the timing prompt information corresponding to the video to be recognized from an information pool corresponding to the timing prompt information. Optionally, the information pool can store pairs of information indices and timing prompt information, where an information index and its pointed-to timing prompt information can specifically be represented as key-value pairs. Furthermore, compared to other timing prompt information in the information pool, the filtered timing prompt information can have the highest similarity between its corresponding information index in the information pool and the video frame features of the video to be recognized.

[0063] S103, determine the recognition result of the video to be recognized based on the semantic prompts, temporal prompts and high-level features corresponding to the video to be recognized.

[0064] Ultimately, the recognition device can fuse (stacking) semantic prompts, temporal prompts, and high-level features of the video to be recognized, thereby utilizing the fused features for video recognition. As mentioned in the background section, this essentially involves recognizing the actions of the target object in the video.

[0065] For video recognition, one optional approach is to input the semantic prompts, temporal prompts, and high-level features of the video to be recognized into the recognition model in the recognition device. The video encoder in the recognition model then further encodes the fused features, and video recognition is achieved based on the encoding results. Optionally, the recognition model can be a pre-trained model, such as a ViT (Vision Transformer) model pre-trained based on ImageNet.

[0066] It should be noted that, following the various usage scenarios mentioned in the background art, the specific manifestation of the identification device in this embodiment and the following embodiments may also be different. For example, for an AR scenario, the identification device may be the AR device itself or a server that provides AR services. The same applies to VR scenarios. For a security scenario, the identification device may be a server matched with a security camera, and the server may store the security video collected by the security camera and further identify the video. Optionally, video identification can also be applied to sports events, for example, for scoring sports movements or judging whether they are standard, etc. In this case, the identification device may be a dedicated video acquisition device for the event.

[0067] In this embodiment, the identification device acquires high-level features of the video to be identified and video frame features of each video frame in the video. The high-level features are used to reflect the overall semantics of the video to be identified, and the video frame features of multiple video frames reflect the temporal correlation of different video frames of the video to be identified, that is, they can reflect the context between different video frames. Then, the identification device determines the semantic prompt information and temporal prompt information corresponding to the video to be identified, and finally realizes the identification of the video to be identified according to the two types of prompt information and the high-level features.

[0068] In the above process, the semantic prompt information and the temporal prompt information are used together to guide the identification direction of the identification device for the video. Compared with images, a video is a kind of dynamic image. Therefore, using two types of prompt information reflecting semantics and context information can provide a more accurate and rich guidance direction for the identification device, thereby enabling the identification device to perform video identification more accurately.

[0069] For the semantic prompt information corresponding to the video to be identified, in addition to Figure 1 the method of determining based on high-level features in the illustrated embodiment, as another optional method, the low-level features of the video to be identified can also be used for determination. Then Figure 2 is a flow chart of a method for determining semantic prompt information provided by an embodiment of the present invention. As shown in Figure 2 , the method may further include the following steps:

[0070] S201, acquiring low-level features of a video to be identified, wherein the low-level features include at least one of spatial features, temporal features and overall features.

[0071] The recognition device first acquires the video to be recognized and encodes it using its configured encoder to obtain low-level features. These low-level features reflect information such as texture, color, edges, and corners of the video. Furthermore, low-level features can include features of at least one dimension, such as spatial features, temporal features, and overall features. Temporal features describe the changes in each video frame over time. Spatial features describe the color features of each video frame, which can also be considered as RGB features. Overall features describe the overall information of the video.

[0072] It should be noted that for video encoding, the recognition device can deploy the same or different encoders. Optionally, different encoders can be used to output high-level and low-level features of the video to be recognized separately. Optionally, the same encoder can also be used to obtain both high-level and low-level features of the video to be recognized. The encoder can include multiple encoding layers; using more encoding layers yields higher-level features.

[0073] S202, determine the semantic prompt information corresponding to the target low-level feature in the low-level features, where the target low-level feature is any one of the low-level features.

[0074] For any one of the low-level features, i.e., the target low-level feature, the recognition device can determine the target index based on the similarity between the target low-level feature and the information index in the information pool. Each low-level feature has an information pool corresponding to that type of feature. Optionally, the target index and the target low-level feature can have the highest similarity. Then, the prompt information corresponding to the target information index in the information pool is determined as the semantic prompt information corresponding to that target low-level feature.

[0075] Among them, the information pool corresponding to the spatial features in the low-level features can be represented as:

[0076] Among them, K S1 K S2 ...K SN For N information indices in the information pool corresponding to spatial features, Each of the aforementioned N information indices points to a semantic prompt, which is represented as a vector with a size of L1*D1, where L1 is the length of the semantic prompt and D1 is the dimension of the spatial feature.

[0077] Similarly, the information pool corresponding to the time features in the low-level features can be represented as:

[0078] Among them, K T1 K T2 ...KTN For N information indices in the information pool corresponding to time features, These are the semantic prompts pointed to by the aforementioned N information indices, and the size of each prompt is L1*D1.

[0079] The information pool corresponding to the overall features can be represented as:

[0080] Among them, K C1 K C2 ...K CN For N information indices in the information pool corresponding to time features, These are the semantic prompts pointed to by the aforementioned N information indices, and the size of each prompt is L1*D1.

[0081] In the manner described above, the recognition device can obtain semantic prompts corresponding to different types of low-level features.

[0082] S203, determine the semantic prompt information corresponding to the video to be identified based on the semantic prompt information corresponding to at least one feature in the low-level features.

[0083] Finally, the semantic prompts corresponding to at least one feature in the lower-level features can be determined as the semantic prompts corresponding to the higher-level features. Optionally, the semantic prompts corresponding to at least one feature can be represented in vector form, and the vectors can be directly added together. The result of the addition can be determined as the semantic prompts corresponding to the video to be identified. In this case, the semantic prompts corresponding to the video to be identified can be represented as follows: in, and These represent semantic prompts of different dimensions obtained from the information pools corresponding to the different types of low-level features mentioned above. i1, i2, and i3 are any integers from 1 to N.

[0084] In this embodiment, prompts can be provided to the recognition device from the perspectives of semantic prompts and contextual information. The semantic prompts are derived from different dimensions of low-level features, and because these different dimensions of semantic prompts are relatively independent and not mixed together, using multi-dimensional semantic prompts can more clearly guide the recognition device in video recognition, thereby further improving the accuracy of video recognition.

[0085] In the above embodiments, the methods for determining semantic and temporal prompt information all utilize information pools. For clarity, the information pool used to determine semantic prompt information can be referred to as the first prompt information pool, and the information pool used to determine temporal prompt information can be referred to as the second prompt information pool. Each low-level feature can have its own corresponding first prompt information pool.

[0086] Based on Figure 2 The method for obtaining semantic prompts shown can be combined with the overall recognition process of the video to be recognized. Figure 3 understand.

[0087] For obtaining timing prompt information, in addition to Figure 1 In the embodiment shown, the timing prompt information corresponding to the information index with the highest similarity is directly determined as the timing prompt information of the video to be identified. Alternatively, the identification device can determine the weight of each information index in the information pool as the similarity between the video frame features and each information index in the information pool, and then comprehensively determine the timing prompt information of the video to be identified based on each information index in the information pool and its weight.

[0088] For example, suppose the prompts corresponding to each information index in the information pool are A, B, C, and D, and the similarities between the video frame features and the above information indices are a, b, c, and d, respectively. Then the temporal prompts corresponding to the video frame features are A*a+B*b+C*c+D*d.

[0089] The information pool corresponding to the video frame features can be represented as: P M ={(K M1 ,P M1 ),(K M2 ,P M2 ),……(K MN ,P MN )}

[0090] Among them, K M1 K M2 ...K MN P represents the N information indices in the information pool corresponding to the video frame features. M1 P M2 ...P MN These are the time-series prompts pointed to by the aforementioned N information indices. The size of the prompts in vector form is L2*D2, where L2 is the length of the time-series prompt and D2 is the dimension of the video frame features.

[0091] In this embodiment, multiple prompts in the information pool are comprehensively considered when determining the timing prompt information, and the similarity is set as a weight to reflect the importance of multiple prompts, so that the final timing prompt information is more comprehensive and can more accurately guide the recognition device to perform video recognition, thereby further improving the accuracy of video recognition.

[0092] Optionally, determining at least one first cue information pool used by the semantic cue information and determining the second cue information pool used by the temporal cue information can both be obtained through pre-training.

[0093] Since the training process for any first cue information pool is the same, we can take the first cue information pool corresponding to the target category feature in the low-level features as an example. The creation process of this information pool can be as follows:

[0094] The recognition device can first acquire a video sample set and extract the low-level features and frame video features of each video sample in the set. The extraction process for low-level features and frame video features can be found in the relevant descriptions in the above embodiments and will not be repeated here. Then, the features of the target type for each video sample are clustered. The cluster centers can be directly determined as the information indexes in the first prompt information pool corresponding to the features of the target type, and initial prompt information is set for each information index. At this point, the first prompt information pool corresponding to the features of the target type is created.

[0095] Similarly, the video frame features of each video sample in the video sample set can be clustered, and the cluster centers can be directly determined as the information indices in the second prompt information pool. At the same time, initial prompt information can be set for each information index. This completes the creation of the second prompt information pool.

[0096] Afterwards, training for both information pools can be completed simultaneously:

[0097] Based on the two information pools created above, the identification device can proceed according to... Figure 1 The illustrated embodiment outputs a recognition result for a video sample based on high-level features of the video sample in the video sample set, initial semantic prompts, and initial temporal prompts. This recognition result can also be considered a potentially biased prediction result, and optionally, this prediction result can be output by a recognition model deployed in the recognition device. In this case, the prediction result is compared with the actual result of the video sample to obtain the result deviation, and this result deviation is used to adjust the initial semantic prompts and initial temporal prompts, thereby achieving training of two information pools. Optionally, the result deviation between the prediction result and the actual result can be calculated using a loss function.

[0098] Among them, it can be according to Figure 2 In the embodiments shown, based on the low-level features of the video samples, initial semantic prompt information is determined in the first prompt information pool corresponding to each type of low-level feature. Alternatively, based on the video frame features of the video samples, initial temporal prompt information corresponding to the video samples can be determined in the second prompt information pool.

[0099] As can be seen, when adjusting the prompt information using the prediction results output from each video sample, only the prompt information used in the output prediction result is adjusted, while other information in the prompt information pool remains unchanged. The above training process can also be combined with... Figure 4 understand.

[0100] Furthermore, since video recognition devices rely heavily on prompts for guidance, the richness and accuracy of these prompts in the information pool significantly impact the recognition model's capabilities, specifically the types of actions it can identify. One approach is to fix the recognition model and improve its capabilities by enriching the prompts in the information pool. In practice, as the number of action types the model needs to recognize increases over time, the prompts in the information pool can be continuously enriched through multiple batches.

[0101] Specifically, at time T1, the common actions of the target object can include the original M types. These M types can be used as the initial task to train a first cue information pool applicable to these M actions, enabling the recognition model to identify the M actions. At time T2, after T1, K new actions of the target object are added. The first cue information pool applicable to these K new actions can then be trained again; that is, the K new actions are used as an incremental task for training the information pool. Finally, by utilizing the cue information from the first cue information pool trained multiple times, the recognition model can be equipped with the ability to identify M+K actions.

[0102] It can be seen that the first cue information pool for this training can be obtained through the incremental task. That is to say, the first cue information pool not only corresponds to a certain type of low-level feature, but also corresponds to a certain training task, that is, to at least one action contained in the training task.

[0103] Optionally, the second cue information pool can be trained simultaneously with the first cue information pool during batch training, resulting in two cue information pools. However, considering that in practice, the contextual information between video frames is not significantly different regardless of whether the video samples contain the original M actions or the newly added K actions, the second cue information pool can also be trained without batch training. Instead, it can be trained at least once with the first cue information pool and then no longer required to be trained.

[0104] Since the first cue information pool corresponding to any low-level feature can be obtained through batch training, the first cue information pool P corresponding to the spatial features mentioned in the above embodiment... S Specifically, it can also be expressed as: in, These are the L first cue information pools corresponding to the spatial features obtained from L batches of training.

[0105] Similarly, the first prompt information pool P corresponding to the time feature T Specifically, it can also be expressed as: The first cue information pool P corresponding to the overall feature C Specifically, it can also be expressed as:

[0106] The above method allows for batch training to obtain a first cue information pool applicable to different actions. Enriching the cue information in this pool improves the recognition model's ability. Compared to directly training the recognition model using video sample sets, this approach offers several advantages. First, since the first cue information pool contains fewer parameters than the recognition model, training with a small number of parameters quickly enhances the model's capabilities. The cue information in the first cue information pool can be considered its parameters. Second, when incremental tasks arise, to prevent catastrophic forgetting, it's necessary to acquire video sample sets corresponding to both incremental and historical tasks. However, if obtaining video sample sets for historical tasks is difficult, the recognition model's ability cannot be improved. Furthermore, because the first cue information pool applicable to different actions is trained in batches—meaning that when training the cue information pool for the newly added K actions, the cue information pool for the original M actions is not used—the recognition device can maintain its ability to recognize the original M actions, preventing catastrophic forgetting. At the same time, when allocating the first cue information pool suitable for new tasks during training, the video sample set corresponding to historical tasks will not be used, thereby improving the above situation and reducing the difficulty of improving the recognition ability of the recognition model.

[0107] Similar to the cue information pool, optionally, when the recognition device uses the recognition model for video recognition, the recognition model can also be pre-trained. This pre-trained recognition model, guided by semantic and temporal cue information selected from the first and second cue information pools respectively, can further encode the high-level features of the video to be recognized, in order to identify different actions of the target object in the video.

[0108] In the above embodiments, the recognition device can perform video recognition using single-modal data (i.e., video modal data). This means it can utilize high-level features of the video to be recognized, as well as semantic and temporal prompts. The specific recognition process can be executed by a recognition model. Specifically, the aforementioned features and prompts can be fused, and the fused features can be input into the recognition model. The recognition model then encodes the fused features and outputs the recognition result of the video to be recognized based on the encoding result.

[0109] Optionally, the recognition device can also use multimodal data (such as video modal data and text modal data) for video recognition. Figure 5 A flowchart illustrating another video recognition method provided in an embodiment of the present invention. Figure 5 As shown, the method may include the following steps:

[0110] S301, Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified.

[0111] S302, determine the timing prompt information corresponding to the video to be identified based on the characteristics of the video frames.

[0112] For the specific implementation process of steps S301 to S302 above, please refer to [link / reference]. Figure 1 The specific descriptions of the relevant steps in the illustrated embodiments will not be repeated here.

[0113] S303, obtain the text features of each of the multiple candidate recognition results corresponding to the video to be recognized.

[0114] S304, Determine the text prompt information corresponding to the text features.

[0115] The alternative recognition results can be various actions that the recognition device can recognize, which can be expressed in text form. The recognition device can then use its own encoder to encode multiple alternative recognition results to obtain various text features of each alternative recognition result.

[0116] Optionally, the recognition device may be equipped with different encoders to encode the video to be recognized and the candidate recognition results separately, so that the high-level features, video frame features and text features of the video to be recognized can be output by different encoders.

[0117] Then, the recognition device can further obtain the text prompt information corresponding to each text feature. Alternatively, the recognition device can determine the text prompt information corresponding to each text feature from a first prompt information pool. According to... Figure 1As described in the illustrated embodiment, the information index and prompt information in the first prompt information pool are stored in the form of key-value pairs. Based on this, one information index in the first prompt information pool can point to a set of prompt information, and a set of prompt information can include different types of prompt information, that is, simultaneously including semantic prompt information and text prompt information. Therefore, when the recognition device follows... Figure 1 or Figure 2 Once the semantic prompt information is determined in the way shown, the text prompt information can be determined simultaneously. Among them, the text prompt information that is the same as the semantic prompt information also belongs to different dimensions.

[0118] When using Figure 2 When determining semantic prompt information in the manner shown, the specific process for determining text prompt information can be as follows: A target index is determined based on the similarity between the target low-level feature and the information index in the first prompt information pool. Here, the target low-level feature is any one of the low-level features, and the first prompt information pool corresponds to the type of target feature. Then, based on the target index corresponding to at least one of the low-level features, the text prompt information corresponding to the text feature is determined in the first prompt information pool.

[0119] It is evident that when specifically adopted Figure 2 When determining semantic prompt information in the manner shown, since the low-level features contain at least one feature and different features belong to different dimensions, the semantic prompt information determined from the first prompt information pool also belongs to different dimensions, such as spatial dimension, time dimension, and overall dimension. Therefore, the determined text prompt information also belongs to different dimensions, that is, the dimension of the text prompt information is the same as the dimension of the semantic prompt information.

[0120] In this embodiment, the first cue information pool corresponding to the spatial features in the low-level features can be represented as:

[0121] Among them, K S1 K S2 ...K SN For N information indices in the information pool corresponding to spatial features, These are the semantic prompts pointed to by the aforementioned N information indices. These are the text prompts pointed to by the aforementioned N information indices. The size of the semantic prompt is L1*D1, where L1 is the length of the semantic prompt and D1 is the dimension of the spatial features; the size of the text prompt is L2*D2, where L2 is the length of the text prompt and D2 is the dimension of the text features.

[0122] Similarly, the information pool corresponding to the time features in the low-level features can be represented as:

[0123] Among them, KT1 K T2 ...K TN For N information indices in the information pool corresponding to time features, These are the semantic prompts pointed to by the aforementioned N information indices. These are the text prompts pointed to by the aforementioned N information indices. The size of the semantic prompt is L1*D1; the size of the text prompt is L2*D2.

[0124] The information pool corresponding to the overall features in the low-level features can be represented as:

[0125] Among them, K C1 K C2 ...K CN These are the N information indices in the information pool corresponding to the overall feature. These are the semantic prompts pointed to by the aforementioned N information indices. These are the text prompts pointed to by the aforementioned N information indices. The size of the semantic prompt is L1*D1; the size of the text prompt is L2*D2.

[0126] S305. Based on the semantic prompts, temporal prompts, text prompts, and high-level features corresponding to the video to be identified, determine the recognition result of the video to be identified.

[0127] Ultimately, the recognition device can use video modal data (i.e., high-level features, semantic prompts, and temporal prompts of the video to be recognized) and text modal data (i.e., text prompts) together to recognize the video to be recognized.

[0128] Optionally, the recognition process in this step can be performed by a recognition model in the recognition device. The recognition model can also be a pre-trained model, such as a contrastive language-image pre-training (CLIP) model.

[0129] Specifically, the recognition model can first fuse the semantic and temporal cue information corresponding to the video to be recognized with high-level features to obtain the video fusion feature V. p =[P vm [V], simultaneously fusing text prompts and text features to obtain text fusion features W. p =[P t ;W].

[0130] Where V represents the high-level features of the video to be identified, and P... vmLet W be the sum of semantic and temporal cues in the video to be identified, and let P be the textual features of the video to be identified. t This is a text prompt message.

[0131] Next, the recognition model inputs video fusion features and text fusion features into the recognition model, so that the recognition model outputs the recognition result of the video to be recognized. Among them, since the text features are obtained by extracting features from a candidate recognition result, the text features correspond to the candidate recognition result, and the text fusion features that incorporate the text features also correspond to the candidate recognition result.

[0132] Optionally, the recognition model may specifically include a first encoder for encoding video and a second encoder for encoding text. Based on this structure, the specific recognition process of the recognition model can be as follows:

[0133] The first encoder in the recognition model extracts features from the video fusion features, while the second encoder extracts features from the text fusion features. Each text fusion feature corresponds to a candidate recognition result, and the feature extracted by the second encoder corresponds to the same candidate recognition result. The recognition model first calculates the similarity between the features extracted by the first encoder and the features extracted by the second encoder, and then uses the feature extracted by the second encoder with the highest similarity as the target feature. The candidate recognition result corresponding to this target feature is then determined as the recognition result for the video to be recognized.

[0134] Furthermore, based on the aforementioned recognition model, the video recognition method provided in this embodiment can also be combined with... Figure 6 understand.

[0135] This embodiment uses multimodal data to achieve video recognition. Details not described in detail here, as well as the technical effects achieved, can be found in the descriptions of the above embodiments and will not be repeated here.

[0136] Optionally, when using multimodal data for video recognition, the first and second cue information pools can also be used, and in conjunction with... Figures 1-4 The difference in the illustrated embodiment is that: in the first cue information pool, one information index points to a set of cue information that simultaneously contains semantic and textual cue information. In this case, the creation and training process of the first cue information pool corresponding to any type of feature (i.e., the target type feature) in the low-level features can be described as follows:

[0137] The recognition device can first acquire a video sample set and extract the low-level features and frame video features of each video sample in the set. The extraction process for low-level features and frame video features can be found in the relevant descriptions in the above embodiments and will not be repeated here. Then, the features of the target type in each video sample are clustered. The cluster centers can be directly determined as the information indices in the first prompt information pool corresponding to the features of the target type. Initial semantic prompt information and initial text prompt information are set for each information index. At this point, the first prompt information pool corresponding to the features of the target type is created.

[0138] The creation process of the second prompt information pool can be found in the description of the relevant embodiments above, and will not be repeated here.

[0139] Afterwards, training for both information pools can be completed simultaneously:

[0140] Based on the information pool created above, the identification device can proceed according to... Figure 5 The illustrated embodiment outputs a recognition result for a video sample based on high-level features of the video sample in the video sample set, initial semantic prompts for the video sample, initial temporal prompts for the video sample, text features of candidate recognition results, and initial text prompts. This recognition result can also be considered a prediction result that may contain biases; optionally, this prediction result can be output by a recognition model deployed in the recognition device.

[0141] At this point, a loss function can be used to calculate the loss between the predicted result and the actual result of the video sample. Based on the calculated loss value, the initial semantic prompts, initial text prompts, and initial temporal prompts input to the recognition model can be adjusted, which means training two information pools.

[0142] Specifically, based on the low-level features of the video samples, initial semantic and textual prompts can be determined simultaneously in the first prompt information pool corresponding to each type of low-level feature. Alternatively, based on the video frame features of the video samples, initial temporal prompts corresponding to the video samples can be determined in the second prompt information pool.

[0143] Furthermore, as can be seen from the above embodiments, the first prompt information pool and the second prompt information pool can be trained in batches. That is, the first prompt information pool and the second prompt information pool corresponding to each batch can be trained using the video sample sets corresponding to different batches.

[0144] During any batch of training, if the action of the target object in each video sample used in this batch of training is Figure 5The multiple alternative recognition results mentioned in the illustrated embodiment can be used to train a first prompt information pool and a second prompt information pool corresponding to the multiple alternative recognition results according to the above process. Optionally, these multiple alternative recognition results can be all or part of the actions of the target object that the recognition device can recognize.

[0145] The training process described above can also improve the recognition model's ability by enriching the cue information in the information pool. Furthermore, the cue information in the information pool can be continuously enriched through multiple batches. Compared to directly using video sample sets to train the recognition model and improve its ability, the above training method offers several advantages. First, since the first cue information pool contains fewer parameters than the recognition model, the recognition model's ability can be quickly improved by training a small number of parameters. Second, because the first cue information pool applicable to different actions is trained in batches, and the cue information pool applicable to the original tasks is not used when training the cue information pool applicable to new tasks, the recognition model's ability to recognize the original M actions is guaranteed, meaning the model will not suffer from catastrophic forgetting. Simultaneously, when obtaining the first cue information pool applicable to new tasks through batch training, the video sample sets corresponding to historical tasks are not used, thus addressing the problem that the difficulty in obtaining video sample sets corresponding to historical tasks hinders the improvement of the recognition model's ability.

[0146] Optionally, while training the first cue information pool in batches, the second cue information pool can also be trained simultaneously, i.e., both cue information pools are trained at the same time. However, considering that in practice, whether the video samples contain the original M types of actions or the video samples contain the newly added K types of actions, the contextual information between video frames is not significantly different, the second cue information pool can also be trained without batch training. Instead, it can be trained at least once with the first cue information pool and then no longer need to be trained.

[0147] The above embodiments have described the process of realizing video recognition from a procedural perspective. Based on this, video recognition can also serve as a service capable of interacting with users. Furthermore, depending on the content of the video, and following the examples in the background art, the service provided to the user can specifically include AR services, VR services, video classification services, security services, sports scoring services, and so on.

[0148] but Figure 8 This is a flowchart illustrating a video recognition method provided in an embodiment of the present invention. The implementing entity of this method can be a service platform. Optionally, this service platform can be deployed in the recognition devices mentioned in the above embodiments. Figure 8 As shown, the method may include the following steps:

[0149] S401, in response to input commands applied to the user interface, acquires the video to be recognized.

[0150] S402, Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified.

[0151] S403, determine the timing prompt information corresponding to the video to be identified based on the characteristics of the video frames.

[0152] S404. Based on the semantic prompts, temporal prompts, and high-level features corresponding to the video to be identified, determine the recognition result of the video to be identified.

[0153] S405 displays the recognition results on the user interface.

[0154] The service platform provides an optional user interface (platform homepage) that can... Figure 9 As shown, on this operation interface, the user can select the service content and input the corresponding video to be recognized. Then, the service platform can execute steps S402 to S405 to display the recognition result on the service platform's operation interface (recognition result page). Taking a security scenario as an example, the recognition result page can display a "no abnormality" recognition result.

[0155] The specific implementation process of each of the above steps can be found in the detailed description of the relevant steps in the above embodiments, and will not be repeated here.

[0156] In this embodiment, the service platform inputs the video to be identified through an interface, and then further obtains the high-level features of the video and the video frame features of each video frame in the video. The high-level features reflect the overall semantics of the video to be identified, while the individual video frame features reflect the temporal relationships between different video frames, i.e., they reflect the context between different video frames. Then, the recognition device determines the semantic and temporal prompts corresponding to the video to be identified, and finally identifies the video based on these two types of prompts and the high-level features. The recognition result can be displayed on the interface provided by the service platform.

[0157] In the above process, semantic and temporal prompts are used together to guide the service platform in its video recognition direction. Compared to images, video is a dynamic image; therefore, utilizing these two types of prompts, which reflect semantic and contextual information, can provide the service platform with more accurate and richer guidance, enabling it to perform video recognition more accurately.

[0158] In addition, other technical effects achieved by this embodiment can be found in the descriptions of the above embodiments, and will not be repeated here.

[0159] When the video recognition methods mentioned in the above embodiments are specifically applied to VR or AR scenarios... Figure 10 This is a flowchart illustrating a feature extraction method provided in an embodiment of the present invention. The execution subject of this method can also be a VR device or an AR device. Figure 10 As shown, the method may include the following steps:

[0160] S501, acquires video containing user actions to be identified.

[0161] S502, Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified.

[0162] S503 determines the timing prompt information corresponding to the video to be identified based on the characteristics of the video frames.

[0163] S504. Based on the semantic prompts, temporal prompts, and high-level features corresponding to the video to be identified, determine the action information contained in the video to be identified.

[0164] S505 displays the image corresponding to the action information.

[0165] Taking a VR scene as an example, the VR device can capture the video to be recognized when the user makes a movement. Then, the VR device can execute the above steps S502 to S505 to make the VR device display the image corresponding to the movement information.

[0166] The specific implementation process of each of the above steps can be found in the detailed description of the relevant steps in the above embodiments, and will not be repeated here.

[0167] In this embodiment, the device acquires a video containing user actions to be identified. It then further obtains high-level features of the video and video frame features of each frame within the video. The high-level features reflect the overall semantics of the video, while the individual video frame features reflect the temporal relationships between different frames, i.e., the context between them. The device then determines the semantic and temporal cues corresponding to the video and ultimately identifies the action information in the video based on these cues and the high-level features, displaying the image corresponding to the action information on the device.

[0168] In the process described above, semantic and temporal cues are used together to guide the device's direction of video recognition. Compared to images, video is a dynamic image; therefore, utilizing these two types of cues, which reflect semantic and contextual information, can provide the device with more accurate and richer guidance, enabling it to perform video recognition more accurately.

[0169] In addition, other technical effects achieved by this embodiment can be found in the descriptions of the above embodiments, and will not be repeated here.

[0170] The following description uses a VR scenario as an example to illustrate the specific implementation process of the video recognition methods provided in the above embodiments.

[0171] When a user uses the VR device at time T1, the scene they see is a cup on a table in a room. At time T2, the user might perform an action including "picking up" the cup. The camera on the VR device can then capture a video containing this action. This video, along with potential recognition results, can then be encoded by different encoders within the VR device to obtain high-level features, low-level features, video frame features, and text features. The potential recognition results can include various actions that the VR device can recognize, such as picking up, direction, walking, chewing, smiling, etc.

[0172] Next, on the one hand, semantic and textual prompts corresponding to different types of low-level features are determined from the first prompt information pool containing different types of low-level features. The sum of multiple semantic prompts is determined as the semantic prompt information corresponding to the video to be identified, and the sum of multiple textual prompts is determined as the textual prompt information corresponding to the text feature. On the other hand, temporal prompts corresponding to the video to be identified are determined from the second prompt information pool.

[0173] Then, the VR device can fuse high-level features, semantic prompts corresponding to the video to be identified, and temporal prompts to obtain video fusion features; it can also fuse the text features of the candidate recognition results and the text prompts corresponding to the text features to obtain text fusion features. These two types of fusion features are then input into a pre-trained model deployed in the VR device, which encodes the video fusion features and text fusion features respectively, and determines the action information contained in the video to be identified based on the encoding results.

[0174] Compared to other alternative recognition results, the pre-trained model determines that the encoding result of the text fusion feature corresponding to the alternative recognition result "pick up" has the highest similarity with the encoding result of the video fusion feature of the video to be recognized. At this time, the model can output the recognition result as "pick up", and the VR device can display the scene of the water glass on the table being picked up to the user at time T3 in response to this recognition result.

[0175] The content of the above scenario embodiments can also be combined with Figure 11 understand.

[0176] When using the above method for video recognition, semantic prompts and temporal prompts are used together to guide the device in the direction of video recognition, thereby enabling the device to perform video recognition more accurately.

[0177] Furthermore, achieving the above recognition relies on training two cue information pools with a relatively small number of parameters, eliminating the need for a pre-trained model with a large number of parameters. Training the cue information pools can also rapidly improve the recognition model's ability. The training processes for the first and second cue information pools can be found in the relevant descriptions in the aforementioned embodiments. Additionally, since the first cue information pool is trained in batches, and the video samples used in different batches are not repeated, this approach avoids catastrophic forgetting in the pre-trained model and also addresses the problem of limited recognition ability due to the difficulty in obtaining video sample sets.

[0178] The following section uses a security scenario as an example to describe the specific implementation process of the video recognition methods provided in the above embodiments.

[0179] Users can access services such as those provided on the service platform. Figure 12 On the homepage shown, select the security service and then input the video to be identified. The service platform can then process the video. In one scenario, the service platform's identification results page may display an identification result of "no abnormality".

[0180] The specific working process of the service platform in this scenario and the technical effects it can achieve can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0181] The following will describe in detail one or more embodiments of the video recognition apparatus of the present invention. Those skilled in the art will understand that these data processing apparatuses can all be configured using commercially available hardware components through the steps taught in this solution.

[0182] Figure 13 This is a schematic diagram of the structure of a video recognition device provided in an embodiment of the present invention, as shown below. Figure 13 As shown, the device includes:

[0183] The first comprehensive feature acquisition module 11 is used to acquire the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified.

[0184] The first timing prompt information determination module 12 is used to determine the timing prompt information corresponding to the video to be identified based on the characteristics of the video frame.

[0185] The first recognition result determination module 13 is used to determine the recognition result of the video to be recognized based on the semantic prompt information corresponding to the video to be recognized, the temporal prompt information, and the high-level features.

[0186] Optionally, the device further includes a low-level feature acquisition module 14, used to acquire low-level features of the video to be identified, wherein the low-level features include at least one of spatial features, temporal features and overall features.

[0187] The semantic prompt information determination module 15 is used to determine the semantic prompt information corresponding to the target low-level feature among the low-level features, wherein the target low-level feature is any one of the low-level features; and to determine the semantic prompt information corresponding to the video to be identified based on the semantic prompt information corresponding to at least one feature among the low-level features.

[0188] Optionally, the semantic prompt information determination module 15 is used to determine a target index based on the similarity between the target low-level feature and the information index in the first prompt information pool, wherein the first prompt information pool corresponds to the category to which the target feature belongs; and to determine the semantic prompt information corresponding to the target low-level feature in the first prompt information pool based on the target index.

[0189] Optionally, the first timing prompt information determination module 12 is used to determine the similarity between the video frame features and each information index in the second prompt information pool as the weight of each information index in the second prompt information pool; and to determine the timing prompt information based on the weight and each prompt information in the second prompt information pool.

[0190] Optionally, the device further includes a text feature acquisition module 16, used to acquire the text features of each of the multiple candidate recognition results corresponding to the video to be recognized.

[0191] The text prompt information determination module 17 is used to determine the text prompt information corresponding to the text feature.

[0192] The first recognition result determination module 13 is used to input the semantic prompt information, the temporal prompt information, the high-level features, the text features and the text prompt information corresponding to the video to be recognized into the recognition model, so that the recognition model outputs the recognition result of the video to be recognized.

[0193] Optionally, the text prompt information determination module 17 is used to determine a target index based on the similarity between the target low-level feature and the information index in the first prompt information pool, wherein the first prompt information pool corresponds to the category to which the target feature belongs; the same information index in the first prompt information pool points to a set of prompt information, the set of prompt information including semantic prompt information and text prompt information; in the first prompt information pool, the text prompt information pointed to by the target index corresponding to at least one low-level feature is determined as the text prompt information corresponding to the text feature.

[0194] Optionally, the first recognition result determination module 13 is used to fuse the semantic prompt information, the temporal prompt information and the high-level features to obtain video fusion features, wherein the temporal prompt information is obtained based on the similarity between the video frame features and each information index in the second prompt information pool; to fuse the text prompt information and the text features to obtain text fusion features; and to input the video fusion features and the text fusion features into the recognition model.

[0195] Optionally, the recognition model includes a first encoder and a second encoder. The first recognition result determination module 13 is further configured to: extract features from the video fusion features using the first encoder; extract features from the text fusion features using the second encoder; determine the similarity between the features extracted by the first encoder and the features extracted by the second encoder; and determine the candidate recognition result corresponding to the target feature extracted by the second encoder as the recognition result of the video to be recognized, wherein the target feature corresponds to the highest similarity.

[0196] Optionally, the apparatus further includes: a creation module 18, configured to acquire low-level features of video samples in a video sample set and video frame features of each video frame in the video samples, wherein the low-level features include at least one of spatial features, temporal features, and overall features, and the video sample set corresponds to a target recognition task; cluster the target type features in the low-level features of the video samples, and determine the cluster center as the information index in a first prompt information pool, wherein the first prompt information pool corresponds to the target type; cluster the video frame features, and determine the cluster center as the information index in a second prompt information pool; and set initial prompt information for the first prompt information pool and the second prompt information pool, respectively.

[0197] Optionally, the apparatus further includes: a training module 19, configured to acquire high-level features, video frame features, and text features of candidate recognition results of video samples in the video sample set; determine initial semantic prompt information and initial text prompt information corresponding to the video sample in the first prompt information pool based on the low-level features of the video sample; determine initial temporal prompt information corresponding to the video sample in the second prompt information pool based on the video frame features of the video sample; input the high-level features, text features, initial semantic prompt information, initial temporal prompt information, and initial text prompt information of the video sample into the recognition model; perform loss calculation based on the recognition result output by the recognition model and the reference result of the video sample; adjust the initial semantic prompt information, initial temporal prompt information, and initial text prompt information based on the calculated loss value to obtain the first prompt information pool and the second prompt information pool corresponding to the plurality of candidate recognition results.

[0198] Figure 13 The device shown can perform Figures 1 to 7 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figures 1 to 7 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figures 1 to 7 The descriptions in the illustrated embodiments will not be repeated here.

[0199] In one possible design, the video recognition methods provided in the above embodiments can be applied to an electronic device, such as... Figure 14 As shown, the electronic device may include a first processor 21 and a first memory 22. The first memory 22 is used to store data supporting the electronic device in performing the above-described actions. Figures 1 to 7 In the video recognition method program provided in the illustrated embodiment, the first processor 21 is configured to execute the program stored in the first memory 22.

[0200] The program includes one or more computer instructions, wherein when executed by the first processor 21, the one or more computer instructions can perform the following steps:

[0201] Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified;

[0202] Based on the video frame features, determine the timing prompt information corresponding to the video to be identified;

[0203] The recognition result of the video to be recognized is determined based on the semantic prompt information corresponding to the video to be recognized, the temporal prompt information, and the high-level features.

[0204] Optionally, the first processor 21 is also used to perform the aforementioned Figures 1 to 7 All or part of the steps in the illustrated embodiments.

[0205] The structure of the electronic device may also include a first communication interface 23 for the electronic device to communicate with other devices or communication systems.

[0206] Figure 15 This is a schematic diagram of another video recognition device provided in an embodiment of the present invention, as shown below. Figure 15 As shown, the device includes:

[0207] The video acquisition module 31 is used to acquire the video to be recognized in response to the input command applied to the operation interface.

[0208] The second feature acquisition module 32 is used to acquire high-level features of the video to be identified and video frame features of video frames in the video to be identified.

[0209] The second timing prompt information determination module 33 is used to determine the timing prompt information corresponding to the video to be identified based on the video frame features.

[0210] The second recognition result determination module 34 is used to determine the recognition result of the video to be recognized based on the semantic prompt information corresponding to the video to be recognized, the temporal prompt information, and the high-level features.

[0211] The recognition result display module 35 is used to display the recognition result on the operation interface.

[0212] Figure 15 The device shown can perform Figures 8 to 9 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figures 8 to 9 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figures 8 to 9 The descriptions in the illustrated embodiments will not be repeated here.

[0213] In one possible design, the video recognition methods provided in the above embodiments can be applied to another electronic device, such as... Figure 16 As shown, the electronic device may include a second processor 41 and a second memory 42. The second memory 42 is used to store data supporting the electronic device in performing the above-described actions. Figures 8-9 In the video recognition program provided in the illustrated embodiment, the second processor 41 is configured to execute the program stored in the second memory 42.

[0214] The program includes one or more computer instructions, wherein the one or more computer instructions, when executed by the second processor 41, can perform the following steps:

[0215] In response to input commands on the user interface, acquire the video to be recognized;

[0216] Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified;

[0217] Based on the video frame features, determine the timing prompt information corresponding to the video to be identified;

[0218] The recognition result of the video to be recognized is determined based on the semantic prompt information corresponding to the video to be recognized, the temporal prompt information, and the high-level features.

[0219] The recognition results are displayed on the user interface.

[0220] Optionally, the second processor 41 is also used to perform the aforementioned Figures 8-9 All or part of the steps in the illustrated embodiments.

[0221] The structure of the electronic device may also include a second communication interface 43 for the electronic device to communicate with other devices or communication systems.

[0222] Figure 17 This is a schematic diagram of the structure of another video recognition device provided in an embodiment of the present invention, as shown below. Figure 17 As shown, the device includes:

[0223] The acquisition module 51 is used to acquire videos containing user actions to be identified.

[0224] The third feature acquisition module 52 is used to acquire high-level features of the video to be identified and video frame features of video frames in the video to be identified.

[0225] The third timing prompt information determination module 53 is used to determine the timing prompt information corresponding to the video to be identified based on the video frame features.

[0226] The action information determination module 54 is used to determine the action information contained in the video to be identified based on the semantic prompt information corresponding to the video to be identified, the temporal prompt information, and the high-level features.

[0227] The action information display module 55 is used to display the interface corresponding to the action information.

[0228] Figure 17 The device shown can perform Figure 10 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figure 10 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figure 10 The descriptions in the illustrated embodiments will not be repeated here.

[0229] In one possible design, the video recognition methods provided in the above embodiments can be applied to another electronic device, such as... Figure 18 As shown, the electronic device may include a third processor 61 and a third memory 62. The third memory 62 is used to store data supporting the electronic device in performing the aforementioned operations. Figure 10 In the video recognition method program provided in the illustrated embodiment, the third processor 61 is configured to execute the program stored in the third memory 62.

[0230] The program includes one or more computer instructions, wherein the one or more computer instructions, when executed by the third processor 61, can perform the following steps:

[0231] Collect videos containing user actions to be identified;

[0232] Obtain high-level features of the video to be identified and video frame features of the video frames in the video to be identified;

[0233] Based on the video frame features, determine the timing prompt information corresponding to the video to be identified;

[0234] Based on the semantic prompt information corresponding to the video to be identified, the temporal prompt information, and the high-level features, the action information contained in the video to be identified is determined;

[0235] Display the interface corresponding to the action information.

[0236] Optionally, the third processor 61 is also used to perform the aforementioned Figure 10 All or part of the steps in the illustrated embodiments.

[0237] The structure of the electronic device may also include a third communication interface 63 for the electronic device to communicate with other devices or communication systems.

[0238] In addition, embodiments of the present invention provide a computer storage medium for storing computer software instructions used by the aforementioned electronic device, which includes instructions for executing the above-mentioned... Figures 1-10 The video recognition method shown involves the following procedures.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video recognition method, characterized in that, include: Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified; Based on the video frame features, determine the timing prompt information corresponding to the video to be identified; Obtain the text features of each of the multiple candidate recognition results corresponding to the video to be recognized; Determine the text prompt information corresponding to the text features; Determine the semantic prompt information corresponding to the target low-level feature in the low-level features of the video to be identified, wherein the target low-level feature is any one of the low-level features, and the low-level feature includes at least one of spatial features, temporal features, and overall features; Based on the semantic prompt information corresponding to at least one of the low-level features, the semantic prompt information corresponding to the video to be identified is determined; The semantic prompt information, the temporal prompt information, the high-level features, the text features, and the text prompt information corresponding to the video to be identified are input into the recognition model, so that the recognition model outputs the recognition result of the video to be identified.

2. The method according to claim 1, characterized in that, The step of determining the semantic prompt information corresponding to the target low-level features in the low-level features of the video to be identified includes: The target index is determined based on the similarity between the target low-level feature and the information index in the first prompt information pool, wherein the first prompt information pool corresponds to the category to which the target low-level feature belongs; Based on the target index, semantic prompt information corresponding to the target low-level feature is determined in the first prompt information pool.

3. The method according to claim 1, characterized in that, The step of determining the timing prompt information of the video to be identified based on the video frame features includes: The similarity between the video frame features and each information index in the second prompt information pool is determined as the weight of each information index in the second prompt information pool; The timing prompt information is determined based on the weights and the prompt information in the second prompt information pool.

4. The method according to claim 1, characterized in that, In the first information pool, the same information index points to a set of information, which includes semantic information and textual information. The step of determining the text prompt information corresponding to the text feature includes: The target index is determined based on the similarity between the target low-level features and the information index in the first prompt information pool, wherein the first prompt information pool corresponds to the category to which the target low-level features belong. In the first prompt information pool, the text prompt information pointed to by the target index corresponding to at least one of the low-level features is determined as the text prompt information corresponding to the text feature.

5. The method according to claim 1, characterized in that, The step of inputting the semantic prompt information, the temporal prompt information, the high-level features, the text features, and the text prompt information corresponding to the video to be identified into the recognition model includes: The semantic prompt information, the temporal prompt information, and the high-level features are fused to obtain video fusion features. The temporal prompt information is obtained based on the similarity between the video frame features and each information index in the second prompt information pool. The text prompt information and the text features are fused together to obtain text fusion features; The video fusion features and text fusion features are input into the recognition model.

6. The method according to claim 5, characterized in that, The recognition model includes a first encoder and a second encoder; The recognition model outputs the recognition result of the video to be recognized, including: The first encoder extracts features from the video fusion features, and the second encoder extracts features from the text fusion features; Determine the similarity between the features extracted by the first encoder and the features extracted by the second encoder; The candidate recognition results corresponding to the target features extracted by the second encoder are determined as the recognition results of the video to be recognized, where the target features correspond to the highest similarity.

7. The method according to claim 4, characterized in that, The method further includes: The low-level features of video samples in the video sample set and the video frame features of each video frame in the video samples are obtained. The low-level features include at least one of spatial features, temporal features and overall features. The video sample set corresponds to the target recognition task. Clustering is performed on the target category features in the low-level features of the video samples, and the cluster center is determined as the information index in the first prompt information pool, which corresponds to the target category. Cluster the video frame features and determine the cluster centers as the information indexes of the second prompt information pool; Initial prompt information is set for the first prompt information pool and the second prompt information pool, respectively.

8. The method according to claim 7, characterized in that, The method further includes: Obtain the high-level features, video frame features, and text features of the candidate recognition results of the video samples in the video sample set; Based on the low-level features of the video sample, the initial semantic prompt information and the initial text prompt information corresponding to the video sample are determined in the first prompt information pool; Based on the video frame features of the video sample, the initial temporal prompt information corresponding to the video sample is determined in the second prompt information pool; The high-level features of the video sample, the text features of the video sample, the initial semantic prompt information, the initial temporal prompt information, and the initial text prompt information of the video sample are input into the recognition model; Loss calculation is performed based on the recognition results output by the recognition model and the reference results of the video samples; The initial semantic prompt information, the initial temporal prompt information, and the initial text prompt information are adjusted based on the calculated loss value to obtain the first prompt information pool and the second prompt information pool corresponding to the multiple candidate recognition results.

9. A video recognition method, characterized in that, include: In response to input commands on the user interface, acquire the video to be recognized; Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified; Based on the video frame features, determine the timing prompt information corresponding to the video to be identified; Obtain the text features of each of the multiple candidate recognition results corresponding to the video to be recognized; Determine the text prompt information corresponding to the text features; Determine the semantic prompt information corresponding to the target low-level feature in the low-level features of the video to be identified, wherein the target low-level feature is any one of the low-level features, and the low-level feature includes at least one of spatial features, temporal features, and overall features; Based on the semantic prompt information corresponding to at least one of the low-level features, the semantic prompt information corresponding to the video to be identified is determined; The semantic prompt information, the temporal prompt information, the high-level features, the text features, and the text prompt information corresponding to the video to be identified are input into the recognition model, so that the recognition model outputs the recognition result of the video to be identified; The recognition results are displayed on the user interface.

10. A video recognition method, characterized in that, Applied to augmented reality (AR) devices or virtual reality (VR) devices, including: Collect videos containing user actions to be identified; Obtain the high-level features of the video to be identified and the video frame features of the video frames in the video to be identified; Based on the video frame features, determine the timing prompt information corresponding to the video to be identified; Obtain the text features of each of the multiple candidate recognition results corresponding to the video to be recognized; Determine the text prompt information corresponding to the text features; Determine the semantic prompt information corresponding to the target low-level feature in the low-level features of the video to be identified, wherein the target low-level feature is any one of the low-level features, and the low-level feature includes at least one of spatial features, temporal features, and overall features; Based on the semantic prompt information corresponding to at least one of the low-level features, the semantic prompt information corresponding to the video to be identified is determined; The semantic prompt information, the temporal prompt information, the high-level features, the text features, and the text prompt information corresponding to the video to be identified are input into the recognition model, so that the recognition model outputs the action information contained in the video to be identified; Display the interface corresponding to the action information.

11. An electronic device, characterized in that, include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the video recognition method as described in any one of claims 1 to 10.

12. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code, which, when executed by a processor of an electronic device, causes the processor to perform the video recognition method as described in any one of claims 1 to 10.

Images