Methods, apparatus, electronic devices and storage media for generating timing action nominations

Abstract

This disclosure relates to a method, apparatus, electronic device, and storage medium for generating temporal action nominations. The method includes: extracting features from multiple video segments obtained from a video to be identified to obtain multiple first features; for any one of the video segments, updating the first features of the video segment based on the correlation between the video segment and its neighboring video segments to obtain second features of the video segment, wherein the neighboring video segments are located within a target neighborhood of the video segment; and generating a target temporal action nomination for the video to be identified based on the second features of the multiple video segments. Embodiments of this disclosure can generate target temporal action nominations with high accuracy corresponding to a video to be identified.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method and apparatus for generating timing action nominations, an electronic device, and a storage medium. Background Technology

[0002] In recent years, with the widespread adoption of short video applications, video temporal action detection has received considerable attention. The main purpose of video temporal action detection is to locate all action segments in an unedited video and determine the action category of each segment; it is a process of localization and recognition. Temporal action detection is a fundamental problem in the field of video understanding and can be well applied to downstream tasks such as video analysis and video action parsing. Most temporal action detection tasks adopt a similar process to object detection, employing a two-stage strategy. First, temporal action nominations are generated for the video; then, based on these nominations, the category of each nomination is determined to obtain the final temporal action detection result. In other words, accurate temporal action nominations are a prerequisite for obtaining high-accuracy temporal action detection results. Summary of the Invention

[0003] This disclosure presents a technical solution for a method and apparatus for generating time-series action nominations, as well as an electronic device and a storage medium.

[0004] According to one aspect of this disclosure, a method for generating a temporal action nomination is provided, comprising: extracting features from multiple video segments obtained from a video to be identified to obtain multiple first features; for any one of the video segments, updating the first features of the video segment based on the association relationship between the video segment and its neighboring video segments to obtain a second feature of the video segment, wherein the neighboring video segments are located within a target neighborhood of the video segment; and generating a target temporal action nomination for the video to be identified based on the second features of the multiple video segments.

[0005] In one possible implementation, updating the first feature of a video segment based on the association between the video segment and its neighboring video segments to obtain the second feature of the video segment includes: for any video segment, extracting temporal contextual information of the first feature of the video segment and its neighboring video segments to obtain the third feature of the video segment; determining the target similarity weight between the video segment and its neighboring video segments; fusing the third feature of the video segment and its neighboring video segments based on the target similarity weight to update the first feature of the video segment; iteratively executing the above steps, and after reaching a preset number of iterations, determining the updated first feature corresponding to the video segment as the second feature of the video segment.

[0006] In one possible implementation, the target neighborhood corresponds to at least one neighborhood scale; determining the target similarity weight between the video segment and its neighboring video segments includes: for any neighborhood scale, determining the initial similarity weight between the video segment and its corresponding neighboring video segments at the neighborhood scale; and fusing the initial similarity weights corresponding to the at least one neighborhood scale to obtain the target similarity weight.

[0007] In one possible implementation, determining the initial similarity weight between the video segment and its corresponding neighboring video segments at the neighborhood scale includes: for any video segment, encoding a third feature of the video segment and its corresponding neighboring video segments at the neighborhood scale to obtain a relationship feature between the video segment and its corresponding neighboring video segments at the neighborhood scale; and decoding the relationship feature to obtain the initial similarity weight between the video segment and its corresponding neighboring video segments at the neighborhood scale.

[0008] In one possible implementation, determining the initial similarity weight between the video segment and its corresponding neighboring video segments at the neighborhood scale includes: for any video segment, determining the third feature of the video segment and the feature distance between the third feature of its corresponding neighboring video segments at the neighborhood scale; and obtaining the initial similarity weight between the video segment and its corresponding neighboring video segments at the neighborhood scale based on the feature distance.

[0009] In one possible implementation, fusing the third features of the video segment and its neighboring video segments based on the target similarity weight to update the first feature of the video segment includes: fusing the third feature of the video segment with the third features of neighboring video segments whose target similarity weight is greater than a similarity weight threshold to update the first feature of the video segment.

[0010] In one possible implementation, generating the target temporal action nomination for the video to be identified based on the second features of the plurality of video segments includes: performing classification processing based on the second features of the plurality of video segments to obtain a first initial temporal action nomination for the video to be identified; performing regression processing based on the second features of the plurality of video segments and the first initial temporal action nomination to obtain a second initial temporal action nomination for the video to be identified; and determining the target temporal action nomination based on the first initial temporal action nomination and the second initial temporal action nomination.

[0011] According to one aspect of this disclosure, a temporal action nomination generation apparatus is provided, comprising: a feature extraction module for extracting features from multiple video segments obtained from a video to be identified, thereby obtaining multiple first features; an update module for updating the first features of any one of the video segments based on the correlation between the video segment and its neighboring video segments, thereby obtaining second features of the video segment, wherein the neighboring video segments are located within a target neighborhood of the video segment; and a nomination generation module for generating a target temporal action nomination for the video to be identified based on the second features of the multiple video segments.

[0012] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0013] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0014] In this embodiment, feature extraction is performed on multiple video segments obtained from the video to be identified to obtain multiple first features. For any given video segment, the first features of the video segment are updated based on the association between the video segment and its neighboring video segments to obtain second features. Based on the second features of multiple video segments, a target temporal action nomination for the video to be identified is generated. By considering only the association between any given video segment and its neighboring video segments within the target neighborhood, local temporal context information is better utilized, the influence of global background noise in the video to be identified is reduced, and the semantic representation capability of the second features of the video segments is improved. This allows for the generation of a target temporal action nomination with high accuracy for the video to be identified based on the second features of multiple video segments.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0017] Figure 1 A flowchart illustrating a method for generating timing action nominations according to an embodiment of the present disclosure is shown;

[0018] Figure 2 A schematic diagram of a timing action nomination generation network according to an embodiment of the present disclosure is shown;

[0019] Figure 3 A schematic diagram of a structured region slot attention module according to an embodiment of the present disclosure is shown;

[0020] Figure 4 A block diagram of a timing action nomination generation apparatus according to an embodiment of the present disclosure is shown.

[0021] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0022] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0024] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0025] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first," "second," and "third" in this document are only used to distinguish different elements and do not constitute a restriction on order.

[0026] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0027] Because video understanding tasks involve more temporal information than ordinary image spatial information, learning the more complex and rich temporal information in videos becomes crucial for temporal action nomination tasks. Some related techniques utilize extended temporal convolutions to improve the temporal receptive field, while others use graph networks to construct explicit temporal connections. However, these methods primarily aim to improve the ability to capture global dependencies in videos, resulting in an inability to fully utilize local temporal contextual information for semantic feature learning. For an unedited video, which contains a large amount of redundant background information, selectively capturing action-related contextual information is essential. Related techniques typically only use self-attention mechanisms to simply learn long-term dependencies without exploring deeper structures to learn better semantic features, leading to low accuracy in generated temporal action nominations.

[0028] This disclosure provides a method for generating temporal action nominations, which can better utilize the contextual semantic information within the target neighborhood based on the structured target neighborhood corresponding to the video segment, reduce the impact of global background noise, improve the learning ability of semantic representation of the video segment, and thus generate target temporal action nominations with high accuracy.

[0029] The following details the process of generating timing action nominations according to embodiments of this disclosure.

[0030] Figure 1 A flowchart illustrating a method for generating timing action nominations according to an embodiment of this disclosure is provided. This method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. This method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server. Figure 1 As shown, the method may include:

[0031] In step S11, feature extraction is performed on multiple video segments obtained from the video to be identified to obtain multiple first features.

[0032] The video to be identified here can be an unedited original video that needs to be used for the generation of temporal action nomination tasks in video temporal action detection. The video to be identified includes multiple video frames continuously acquired by the image acquisition device in the time dimension.

[0033] Based on a preset segment length, the video to be recognized is segmented into multiple video segments. For example, if the video to be recognized includes T video frames arranged in chronological order, and the preset segment length is σ frames, then each consecutive σ frames in time will be segmented, and one frame will be randomly selected from each consecutive σ frames after segmentation as a video segment. Finally, L = T / σ video segments are obtained, and each video segment is a video frame. The specific values ​​of T and σ can be determined according to the actual situation, and this disclosure does not impose specific limitations on them.

[0034] Feature extraction is performed on multiple video segments to obtain the first feature of each video segment.

[0035] In one example, a video recognition network (e.g., I3D, Inflated 3D ConvNet) can be used to extract features from multiple video segments to obtain initial features for each video segment. The initial features can be in the form of feature maps, and the feature dimension of the initial features is C.

[0036] For any video segment, a linear transformation is performed on the initial features of the video segment to obtain the first feature of the video segment. The first feature can be in the form of a feature vector, and the feature dimension of the first feature is C. input .

[0037] In step S12, for any video segment, based on the association between the video segment and its neighboring video segments, the first feature of the video segment is updated to obtain the second feature of the video segment, wherein the neighboring video segments are located within the target neighborhood of the video segment.

[0038] For any given video segment, attention learning is performed on the video segment and its neighboring video segments within the target neighborhood based solely on a structured target neighborhood. This allows for updating the first feature of the video segment based on local contextual information, resulting in a second feature with semantic representation capabilities corresponding to the video segment.

[0039] The following sections will describe in detail the specific process of updating the first feature of a video segment to obtain its second feature, based on the possible implementation methods disclosed herein; these details will not be elaborated upon here.

[0040] In step S13, a target temporal action nomination for the video to be identified is generated based on the second features of multiple video segments.

[0041] Since the second feature of each video segment has a high semantic representation capability, based on the second features of multiple video segments, a target temporal action nomination with high accuracy corresponding to the video to be identified can be generated.

[0042] The following sections will describe in detail the specific process of generating target temporal action nominations for the video to be identified based on the second features of multiple video segments, in conjunction with the possible implementation methods disclosed herein. These details will not be elaborated upon here.

[0043] According to the embodiments of this disclosure, for any video segment corresponding to the video to be identified, only the association relationship between it and the neighboring video segments in the target neighborhood is considered. This makes better use of the local context information in the time sequence, reduces the influence of global background noise in the video to be identified, and improves the semantic representation ability of the second feature of the video segment. This enables the generation of a target temporal action nomination with high accuracy corresponding to the video to be identified based on the second features of multiple video segments.

[0044] In one example, a temporal action nomination generation network can be used to generate the aforementioned temporal action nominations. Figure 2 A schematic diagram of a timing action nomination generation network according to an embodiment of the present disclosure is shown. Figure 2 As shown, the temporal action nomination generation network includes a feature extraction module. Multiple video segments obtained by segmenting the video to be identified are input into the temporal action nomination generation network, and the feature extraction module is used to extract features from the multiple video segments to obtain multiple first features.

[0045] For example, the video to be recognized is input into a temporal action nomination generation network. After feature extraction by the feature extraction module, the first feature of each of the L video segments obtained from the video to be recognized is obtained. The first feature of any video segment can be represented as... The dimension of each first feature is C. input Then the feature extraction module outputs the L×C value corresponding to the video to be identified. input The first characteristic sequence of size.

[0046] In one possible implementation, based on the correlation between a video segment and its neighboring video segments, the first feature of the video segment is updated to obtain the second feature of the video segment. This includes: for any video segment, extracting the temporal context information of the first feature of the video segment and its neighboring video segments to obtain the third feature of the video segment; determining the target similarity weight between the video segment and its neighboring video segments; fusing the third feature of the video segment and its neighboring video segments based on the target similarity weight to update the first feature of the video segment; iteratively executing the above steps, and when a preset number of iterations is reached, determining the updated first feature corresponding to the video segment as the second feature of the video segment.

[0047] For any given video segment, only the correlation between it and its neighboring video segments in the target neighborhood is considered. This makes better use of local contextual information in time, reduces the risk of overfitting during updates, reduces the impact of global background noise, and improves the semantic representation ability of the second feature of the final video segment through multiple iterative updates.

[0048] In one possible implementation, the target neighborhood corresponds to at least one neighborhood scale; determining the target similarity weight between a video segment and its neighboring video segments includes: for any neighborhood scale, determining the initial similarity weight between the video segment and its corresponding neighboring video segments at that neighborhood scale; and fusing the initial similarity weights corresponding to at least one neighborhood scale to obtain the target similarity weight.

[0049] Since the duration of different video actions in the video to be identified varies greatly, during an iterative update, for any given video segment, the association between it and the neighboring video segments in multiple target neighborhoods corresponding to different neighborhood scales is determined. This makes better use of different local contextual information in time, so that in this iterative update, a more accurate target similarity weight is obtained between the video segment and the neighboring video segments in the target neighborhood with the largest neighborhood scale.

[0050] As mentioned above Figure 2 For example, Figure 2 As shown, the temporal action nomination generation network includes a structured region slot attention module. The L×C value corresponding to the video to be identified, output by the feature extraction module, is then used. input The first feature sequence of size is input into the structured region slot attention module, and the structured region slot attention module is used to perform T iterations to update and output the second feature of each video segment.

[0051] Figure 3 A schematic diagram of a structured region slot attention module according to an embodiment of the present disclosure is shown. During the t-th iteration update, contextual information is extracted temporally for the first feature of each video segment after the (t-1)-th iteration update to obtain the third feature of the video segment for the t-th iteration update.

[0052] In one example, for any video segment, based on temporal convolution and nonlinear processing, the temporal contextual information of the first feature of the video segment and its neighboring video segments is extracted to obtain the third feature of the video segment.

[0053] For the i-th video segment, the first feature of the i-th video segment after the (t-1)-th iteration is updated. Temporal convolution (Convs) and nonlinear (ReLU) processing are performed to obtain the third feature of the i-th video segment. The third feature The query feature Q and value feature V are the corresponding features for the i-th video segment.

[0054] When the neighborhood scale is s, the target neighborhood corresponding to the i-th video segment includes: the is-th video segment, ..., the (i-1)-th video segment, the i-th video segment, the (i+1)-th video segment, and the (i+s)-th video segment. Therefore, the third feature of the neighboring video segments in the target neighborhood corresponding to the i-th video segment at neighborhood scale s is determined as the key feature K(Key) corresponding to the i-th video segment. That is, the key feature K of the i-th video segment can be expressed as...

[0055] In one example, the structured region slot attention module pre-sets multiple neighborhood scales {s1,s2,...,s}. N At this point, the i-th video segment corresponds to multiple key features K at different neighborhood scales:

[0056] For example, the structured region slot attention module, based on multiple hierarchical local regions, pre-sets multiple neighborhood scales {1,2}. In this case, the i-th video segment corresponds to key features K at two different neighborhood scales:

[0057] Based on the attention mechanism, for the i-th video segment, the association between the query feature Q corresponding to the i-th video segment and the key feature K at each neighborhood scale is determined, that is, the initial similarity weight between the i-th video segment and its corresponding neighboring video segments at each neighborhood scale is determined.

[0058] Taking the i-th video segment as an example again, determine the query feature Q corresponding to the i-th video segment: Key features K of the i-th video segment at a neighborhood scale s=1: Initial similarity weights between Includes: the third feature of the i-th video segment and the third feature of the (i-1)th video segment Initial similarity weights between The third feature of the i-th video segment and the third feature of the (i+1)th video segment Initial similarity weights between

[0059] Determine the query feature Q corresponding to the i-th video segment: Key features K of the target neighborhood corresponding to the i-th video segment at a neighborhood scale s=2: Initial similarity weights between Includes: the third feature of the i-th video segment and the third feature of the (i-2)th video segment Initial similarity weights between The third feature of the i-th video segment and the third feature of the (i-1)th video segment Initial similarity weights between The third feature of the i-th video segment and the third feature of the (i+1)th video segment Initial similarity weights between The third feature of the i-th video segment and the third feature of the (i+2)th video segment Initial similarity weights between

[0060] The initial similarity weights of the i-th video segment at multiple neighborhood scales are fused to obtain the target similarity weight of the i-th video segment in the target neighborhood with the largest neighborhood scale.

[0061] Taking the i-th video segment as an example again, the initial similarity weights of the i-th video segment at two neighborhood scales s=1 and s=2 are fused to obtain the target similarity weight of the i-th video segment in the target neighborhood at a neighborhood scale s=2. Includes: the third feature of the i-th video segment and the third feature of the (i-2)th video segment Target similarity weights between The third feature of the i-th video segment and the third feature of the (i-1)th video segment Target similarity weights between The third feature of the i-th video segment and the third feature of the (i+1)th video segment Target similarity weights between The third feature of the i-th video segment and the third feature of the (i+2)th video segment Target similarity weights between

[0062] Based on the target similarity weight of the i-th video segment in the target neighborhood The third features of multiple video segments within the target neighborhood are fused to update the first feature of the i-th video segment, resulting in the updated first feature of the i-th video segment after the t-th iteration.

[0063] Taking the i-th video segment as an example again, the first feature corresponding to the i-th video segment after the t-th iteration update...

[0064] Similarly, in the t-th iteration, the first feature of each video segment is updated in the manner described above. This process is repeated iteratively until a preset number of iterations is reached, and the updated first feature of each video segment after reaching the preset number of iterations is then determined as its second feature.

[0065] In one possible implementation, for any neighborhood scale, determining the initial similarity weight between a video segment and its corresponding neighboring video segments at that neighborhood scale includes: for any video segment, encoding the third feature of the video segment and its corresponding neighboring video segments at that neighborhood scale to obtain the relationship feature between the video segment and its corresponding neighboring video segments at that neighborhood scale; and decoding the relationship feature between the video segment and its corresponding neighboring video segments at that neighborhood scale to obtain the initial similarity weight between the video segment and its corresponding neighboring video segments at that neighborhood scale.

[0066] As mentioned above Figure 3 For example, Figure 3 As shown, the structured region slot attention module includes an encoder and a decoder. The encoder encodes multiple third features to obtain the target relation feature map.

[0067] L third features are input into the encoder. The encoder is used to perform dimensional transformation on the L third features to construct an initial relation feature map of size L×L. The i-th row and j-th column of the initial relation feature map is used to indicate the initial relation features between the i-th video segment and the j-th video segment.

[0068] Based on a 2D convolutional layer with kernel size (2s+1,1), padding size (s,0), ​​and stride size (1,1), the initial relational feature map is processed by 2D convolution. That is, local context information of size 2s+1 is obtained in each column of the 2D feature map, resulting in a dense relational feature map F' of size L×L.

[0069] Since the dense relationship feature map F' contains many interference terms, in order to obtain only the association between each video segment and multiple neighboring video segments in its target neighborhood, the dense relationship feature map F' is filtered based on the neighborhood scale s of the target neighborhood and the following formula (1) to obtain the target relationship feature map F'.

[0070]

[0071] Among them, F” i,j This represents the feature relationship between the third feature of the i-th video segment and the j-th video segment.

[0072] For the i-th video segment, the target feature relationship graph includes: the third feature of the i-th video segment. And the third feature of the two neighboring video segments (the (i-1)th and (i+1)th video segments in the target neighborhood corresponding to a neighborhood scale s=1. The characteristic relationship between them: F i,i-1 F i,i+1 .

[0073] Using a decoder, the target relationship feature map corresponding to each neighborhood scale is decoded to obtain the initial similarity weight for each neighborhood scale. When the neighborhood scale s = 1, the target relationship feature map corresponding to neighborhood scale s = 1 is decoded to obtain the initial similarity weight for neighborhood scale s = 1.

[0074] In one possible implementation, for any neighborhood scale, determining the initial similarity weight between a video segment and its corresponding neighboring video segments at that neighborhood scale includes: for any video segment, determining the third feature of the video segment and the feature distance between the third feature of its corresponding neighboring video segments at that neighborhood scale; and obtaining the initial similarity weight between the video segment and its corresponding neighboring video segments at that neighborhood scale based on the feature distance.

[0075] By determining the third feature of a video segment and the feature distance between it and the third feature of its neighboring video segments at any neighborhood scale, the initial similarity weight between the video segment and its neighboring video segments at any neighborhood scale can be quickly determined.

[0076] Taking the i-th video segment as an example again, the query feature Q corresponding to the i-th video segment is determined as follows: Key features K of the i-th video segment in a neighborhood of scale s = 2: Initial similarity weights between Time: This can be achieved by determining the third feature of the i-th video segment. and the third feature of the (i-2)th video segment The feature distance between them is used as the initial similarity weight. By determining the third feature of the i-th video segment and the third feature of the (i-1)th video segment The feature distance between them is used as the initial similarity weight. By determining the third feature of the i-th video segment and the third feature of the (i+1)th video segment The feature distance between them is used as the initial similarity weight. By determining the third feature of the i-th video segment and the third feature of the (i+2)th video segment The feature distance between them is used as the initial similarity weight.

[0077] The method for determining the feature distance between two features can be the cosine distance algorithm, the Euclidean distance algorithm, the dot product distance algorithm, etc., and this disclosure does not make a specific limitation on it.

[0078] In one possible implementation, the third features of a video segment and its neighboring video segments are fused based on the target similarity weight to update the first feature of the video segment. This includes fusing the third feature of the video segment with the third features of neighboring video segments whose target similarity weight is greater than a similarity weight threshold to update the first feature of the video segment.

[0079] To better utilize the local context in time, a similarity weight threshold is set, and only the third feature of the video segment and the third features of neighboring video segments whose target similarity weight is greater than the similarity weight threshold are fused to update the first feature of the video segment.

[0080] Taking the t-th iteration update process corresponding to the i-th video segment as an example, the target similarity weight of the i-th video segment in the target neighborhood... include: In target similarity weight and If the similarity weight is less than the similarity weight threshold, only the target similarity weight is applied. and When the similarity weight threshold is greater than the threshold, the first feature of the i-th video segment after the t-th iteration update.

[0081] In one possible implementation, generating a target temporal action nomination for the video to be identified based on the second features of multiple video segments includes: performing classification processing based on the second features of multiple video segments to obtain a first initial temporal action nomination for the video to be identified; performing regression processing based on the second features of multiple video segments and the first initial temporal action nomination to obtain a second initial temporal action nomination for the video to be identified; and determining a target temporal action nomination based on the first initial temporal action nomination and the second initial temporal action nomination.

[0082] Based on the above description, each video segment is a video frame in a video to be identified. The second features of multiple video segments are then classified to obtain the initial prediction confidence that each video segment is the starting frame of an action, and the ending prediction confidence that each video segment is the ending frame of an action.

[0083] Then, based on the initial confidence threshold and the final confidence threshold, multiple video frames for the start of a prediction action (initial prediction confidence greater than the initial confidence threshold) and multiple video frames for the end of a prediction action (final prediction confidence greater than the final confidence threshold) are determined.

[0084] Based on a combination of any predicted action start video frame and any predicted action end video frame, multiple first initial temporal action nominations are determined for the video to be identified. For example, the first initial temporal action nominations include: First initial temporal action nomination A: the 3rd video segment (predicted action start video frame) and the 6th video segment (predicted action end video frame); First initial temporal action nomination B: the 4th video segment (predicted action start video frame) and the 7th video segment (predicted action end video frame).

[0085] As mentioned above Figure 2 For example, Figure 2 As shown, the temporal action nomination generation network includes a nomination classifier. The nomination classifier is used to classify the second features of multiple video segments to obtain the first initial temporal action nomination for the video to be identified.

[0086] For any first initial temporal action nomination, determine the nomination features of that nomination. For example, for the aforementioned first initial temporal action nomination A: the 3rd video segment (predicted action start video frame) and the 6th video segment (predicted action end video frame), determine the nomination features of the first initial temporal action nomination A based on the second features of the 3rd to 6th video segments. In one example, the average of the four second features corresponding to the 3rd to 6th video segments can be used as the nomination features of the first initial temporal action nomination A. Similarly, the nomination features of the second initial temporal action nomination B are obtained.

[0087] Regression processing is performed on the nomination features of the first initial temporal action nomination to obtain the second initial temporal action nomination and its regression confidence. The number of second initial temporal action nominations is the same as the number of first initial temporal action nominations.

[0088] For example, regression processing is performed on the nomination features of the first initial temporal action nomination A to obtain the second initial temporal action nomination A': the third video segment (predicted action start video frame) and the sixth video segment (predicted action end video frame), as well as the regression confidence of the second initial temporal action nomination A'.

[0089] Regression processing is performed on the nomination features of the first initial temporal action nomination B to obtain the second initial temporal action nomination B': the third video segment (predicted action start video frame) and the sixth video segment (predicted action end video frame), as well as the regression confidence of the second initial temporal action nomination B'.

[0090] As mentioned above Figure 2 For example, Figure 2 As shown, the temporal action nomination generation network includes a nomination regressor. Based on the second features of multiple video segments and the first initial temporal action nomination, the nomination regressor performs regression processing to obtain the second initial temporal action nomination of the video to be identified.

[0091] The first and second initial temporal action nominations with the same starting point are fused. For example, the first initial temporal action nomination A and the second initial temporal action nomination A' are fused to obtain the fused initial temporal action nomination A'", and its target confidence is equal to the regression confidence of the second initial temporal action nomination A' × the initial prediction confidence of the first initial temporal action nomination A × the ending prediction confidence of the first initial temporal action nomination A.

[0092] Similarly, by fusing the first initial temporal action nomination B and the second initial temporal action nomination B', we obtain the fused initial temporal action nomination B', and its target confidence is equal to the regression confidence of the second initial temporal action nomination B' × the initial prediction confidence of the first initial temporal action nomination B × the final prediction confidence of the first initial temporal action nomination B.

[0093] Using a target confidence threshold, the initial temporal action nominations after fusion are filtered, and the initial temporal action nominations after fusion with a target confidence greater than the target confidence threshold are identified as target temporal action nominations.

[0094] To further improve the accuracy of the nomination, the target temporal action nomination can be further processed by non-maximum suppression (NMS) to obtain the target temporal action nomination of the video to be identified.

[0095] In this embodiment, feature extraction is performed on multiple video segments obtained from the video to be identified to obtain multiple first features. For any given video segment, the first features of the video segment are updated based on the association between the video segment and its neighboring video segments to obtain second features. Based on the second features of multiple video segments, a target temporal action nomination for the video to be identified is generated. By considering only the association between any given video segment and its neighboring video segments within the target neighborhood, local temporal context information is better utilized, the influence of global background noise in the video to be identified is reduced, and the semantic representation capability of the second features of the video segments is improved. This allows for the generation of a target temporal action nomination with high accuracy for the video to be identified based on the second features of multiple video segments.

[0096] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0097] In addition, this disclosure also provides a timing action nomination generation apparatus, electronic device, computer-readable storage medium, and program, all of which can be used to implement any of the timing action nomination generation methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding records in the method section and will not be repeated here.

[0098] Figure 4 A block diagram of a timing action nomination generation apparatus according to an embodiment of the present disclosure is shown. Figure 4 As shown, the device 40 includes:

[0099] Feature extraction module 41 is used to extract features from multiple video segments obtained from the video to be identified, and obtain multiple first features;

[0100] Update module 42 is used to update the first feature of any video segment based on the relationship between the video segment and its neighboring video segments to obtain the second feature of the video segment, wherein the neighboring video segments are located within the target neighborhood of the video segment;

[0101] The nomination generation module 43 is used to generate target temporal action nominations for the video to be identified based on the second features of multiple video segments.

[0102] In one possible implementation, update module 42 includes:

[0103] The first determining submodule is used to extract the temporal context information of the first feature of the video segment and its neighboring video segments for any video segment, so as to obtain the third feature of the video segment.

[0104] The second determination submodule is used to determine the target similarity weight between the video segment and its neighboring video segments;

[0105] The third determining submodule is used to fuse the third features of the video segment and its neighboring video segments based on the target similarity weights between the video segment and its neighboring video segments, so as to update the first features of the video segment.

[0106] The iterative submodule is used to iteratively execute the above steps. When the preset number of iterations is reached, the updated first feature corresponding to the video segment is determined as the second feature of the video segment.

[0107] In one possible implementation, the target neighborhood corresponds to at least one neighborhood scale;

[0108] The second determination submodule includes:

[0109] The first determining unit is used to determine the initial similarity weight between the video segment and its neighboring video segments at any neighborhood scale.

[0110] The second determining unit is used to fuse the initial similarity weights corresponding to at least one neighborhood scale to obtain the target similarity weights.

[0111] In one possible implementation, the first determining unit is specifically used for:

[0112] For any video segment, the third feature of the video segment and its neighboring video segments at the corresponding neighborhood scale is encoded to obtain the relationship feature between the video segment and its neighboring video segments at the corresponding neighborhood scale.

[0113] The relationship features between the video segment and its neighboring video segments at the same neighborhood scale are decoded to obtain the initial similarity weights between the video segment and its neighboring video segments at the same neighborhood scale.

[0114] In one possible implementation, the first determining unit is specifically used for:

[0115] For any video segment, determine the third feature of the video segment and the feature distance between it and the third feature of its neighboring video segments at the corresponding neighborhood scale;

[0116] Based on the feature distance, the initial similarity weight between the video segment and its neighboring video segments at that neighborhood scale is obtained.

[0117] In one possible implementation, the third determining submodule is specifically used for:

[0118] The third feature of the video segment is fused with the third feature of neighboring video segments whose target similarity weight is greater than the similarity weight threshold to update the first feature of the video segment.

[0119] In one possible implementation, the nomination generation module 43 is specifically used for:

[0120] Based on the second features of multiple video segments, classification processing is performed to obtain the first initial temporal action nomination of the video to be identified.

[0121] Based on the second features of multiple video segments and the first initial temporal action nomination, regression processing is performed to obtain the second initial temporal action nomination of the video to be identified.

[0122] The target timing action nomination is determined based on the first initial timing action nomination and the second initial timing action nomination.

[0123] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0124] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0125] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0126] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0127] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0128] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0129] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 5 Electronic device 800 can be user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device and other terminal devices.

[0130] Reference Figure 5 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0131] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0132] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0133] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0134] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0135] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0136] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0137] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0138] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G, LTE, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID), Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0139] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0140] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0141] This disclosure relates to the field of augmented reality (AR). It involves acquiring image information of target objects in a real-world environment and then using various visual algorithms to detect or identify the relevant features, states, and attributes of these objects, thereby achieving an AR effect that combines virtual and real elements to suit specific applications. For example, target objects may include human features such as faces, limbs, gestures, and movements; objects such as signs and markers; or venues such as sand tables, display areas, or displayed items. Visual algorithms may include visual localization, SLAM, 3D reconstruction, image registration, background segmentation, keypoint extraction and tracking of objects, and pose or depth detection. Specific applications can include interactive scenarios related to real-world scenes or objects, such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays, as well as human-related special effects processing, such as makeup enhancement, limb enhancement, special effects displays, and virtual model displays. Convolutional neural networks (CNNs) can be used to detect or identify the relevant features, states, and attributes of target objects. The aforementioned CNNs are network models trained using deep learning frameworks.

[0142] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 6 The electronic device 1900 can be provided as a server or a terminal device. (See reference...) Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0143] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM(or similar.)

[0144] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0145] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0146] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0147] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0148] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0149] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0151] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0153] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0154] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0155] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0156] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0157] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating temporal action nominations, characterized in that, include: Feature extraction is performed on multiple video segments obtained from the video to be identified to obtain multiple first features; For any of the video segments, based on the association between the video segment and its neighboring video segments, the first feature of the video segment is updated to obtain the second feature of the video segment, wherein the neighboring video segments are located within the target neighborhood of the video segment; Based on the second features of the multiple video segments, a target temporal action nomination for the video to be identified is generated. The step of updating the first feature of the video segment based on the correlation between the video segment and its neighboring video segments to obtain the second feature of the video segment includes: For any of the video segments, extract the temporal context information of the first features of the video segment and its neighboring video segments to obtain the third feature of the video segment; Determine the target similarity weights between the video segment and its neighboring video segments; Based on the target similarity weight, the third features of the video segment and its neighboring video segments are fused to update the first feature of the video segment; The above steps are executed iteratively. When the preset number of iterations is reached, the updated first feature corresponding to the video segment is determined as the second feature of the video segment.

2. The method according to claim 1, characterized in that, The target neighborhood corresponds to at least one neighborhood scale; Determining the target similarity weight between the video segment and its neighboring video segments includes: For any of the neighborhood scales, determine the initial similarity weights between the video segment and its corresponding neighboring video segments at the neighborhood scale; The initial similarity weights corresponding to the at least one neighborhood scale are fused to obtain the target similarity weight.

3. The method according to claim 2, characterized in that, Determining the initial similarity weight between the video segment and its corresponding neighboring video segments at the neighborhood scale includes: For any of the video segments, the third feature of the video segment and its neighboring video segments at the neighborhood scale is encoded to obtain the relationship feature between the video segment and its neighboring video segments at the neighborhood scale. The relationship features are decoded to obtain the initial similarity weights between the video segment and its neighboring video segments at the neighborhood scale.

4. The method according to claim 2, characterized in that, Determining the initial similarity weight between the video segment and its corresponding neighboring video segments at the neighborhood scale includes: For any of the video segments, determine the third feature of the video segment and the feature distance between it and the third feature of its neighboring video segments at the neighborhood scale; Based on the feature distance, the initial similarity weight between the video segment and its neighboring video segments at the neighborhood scale is obtained.

5. The method according to any one of claims 1 to 4, characterized in that, the step of fusing the third features of the video segment and its neighboring video segments based on the target similarity weight to update the first feature of the video segment includes: The third feature of the video segment and the third feature of neighboring video segments whose target similarity weight is greater than the similarity weight threshold are fused together to update the first feature of the video segment.

6. The method according to any one of claims 1 to 4, characterized in that, The step of generating a target temporal action nomination for the video to be identified based on the second features of the multiple video segments includes: Based on the second features of the multiple video segments, classification processing is performed to obtain the first initial temporal action nomination of the video to be identified; Based on the second features of the multiple video segments and the first initial temporal action nomination, regression processing is performed to obtain the second initial temporal action nomination of the video to be identified; The target timing action nomination is determined based on the first initial timing action nomination and the second initial timing action nomination.

7. A device for generating temporal action nominations, characterized in that, include: The feature extraction module is used to extract features from multiple video segments obtained from the video to be identified, and obtain multiple first features; An update module is configured to update a first feature of any video segment based on the association between the video segment and its neighboring video segments to obtain a second feature of the video segment, wherein the neighboring video segments are located within the target neighborhood of the video segment; The nomination generation module is used to generate target temporal action nominations for the video to be identified based on the second features of the plurality of video segments; The update module includes: The first determining submodule is used to extract the temporal context information of the first feature of the video segment and its neighboring video segments for any given video segment, so as to obtain the third feature of the video segment; The second determining submodule is used to determine the target similarity weight between the video segment and its neighboring video segments; The third determining submodule is used to fuse the third features of the video segment and its neighboring video segments based on the target similarity weight, so as to update the first feature of the video segment; The iterative submodule is used to iteratively execute the above steps. When a preset number of iterations is reached, the updated first feature corresponding to the video segment is determined as the second feature of the video segment.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images