A method, apparatus and device for determining characteristics of an object

Abstract

The application provides a method, device and equipment for determining object features, comprising: obtaining a plurality of target object features corresponding to key frames in a sample video, and obtaining a text feature corresponding to a sample text; determining a spatial positioning score and a time positioning score between the text feature and the key frames based on the text feature and the plurality of target object features; determining a first loss value based on the spatial positioning score and the time positioning score; selecting a representative object feature from the plurality of target object features corresponding to the key frames, and determining a feature effect score corresponding to the representative object feature; selecting a representative key frame from the plurality of key frames of the sample video, and determining a frame effect score corresponding to the representative key frame; determining a second loss value based on the feature effect score and the frame effect score; determining a target loss value based on the first loss value and the second loss value, and obtaining a target network model based on the target loss value. Through the application scheme, the training time of the target network model is short, and the resource consumption is small.

Description

Technical Field

[0001] This application relates to the field of video processing, and in particular to a method, apparatus, and device for determining object features. Background Technology

[0002] With the continuous development of internet technology, video has become one of the most popular media. Faced with massive amounts of video content, accurately retrieving objects of interest from videos is increasingly important. To accurately retrieve these objects, a network model can be trained to detect them. However, training this model requires fine-grained spatiotemporal annotation of objects in the sample videos—for example, annotating the frame containing the object and its location within that frame. Clearly, this requires significant manual effort, consuming substantial server resources and time. Consequently, the annotation process is lengthy, slow, and resource-intensive. Summary of the Invention

[0003] This application provides a method for determining the features of an object, the method comprising:

[0004] Obtain multiple target object features corresponding to keyframes in the sample video, and obtain text features corresponding to sample text in the sample video; based on the text features and the multiple target object features, determine the spatial localization score and temporal localization score between the text features and the keyframes; based on the spatial localization score and the temporal localization score, determine the first loss value corresponding to the sample video;

[0005] For each keyframe in the sample video, a representative object feature is selected from multiple target object features corresponding to that keyframe, and the feature effect score corresponding to the representative object feature is determined; a representative keyframe is selected from multiple keyframes in the sample video, and the frame effect score corresponding to the representative keyframe is determined; a second loss value corresponding to the sample video is determined based on the feature effect score and the frame effect score.

[0006] A target loss value is determined based on a first loss value and a second loss value. A target network model is trained based on the target loss value. The target network model is used to determine the object features corresponding to the video to be detected.

[0007] This application provides a device for determining the features of an object, the device comprising:

[0008] The acquisition module is used to acquire multiple target object features corresponding to key frames in the sample video, and to acquire text features corresponding to sample text in the sample video.

[0009] The determination module is configured to determine the spatial localization score and temporal localization score between the text features and the keyframes based on the text features and the multiple target object features; and to determine the first loss value corresponding to the sample video based on the spatial localization score and the temporal localization score.

[0010] The determining module is further configured to, for each keyframe in the sample video, select representative object features from multiple target object features corresponding to the keyframe, and determine the feature effect score corresponding to the representative object features; select representative keyframes from multiple keyframes in the sample video, and determine the frame effect score corresponding to the representative keyframes; determine a second loss value corresponding to the sample video based on the feature effect score and the frame effect score; and determine a target loss value based on the first loss value and the second loss value.

[0011] The processing module is used to train a target network model based on the target loss value, wherein the target network model is used to determine the object features corresponding to the video to be detected.

[0012] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; wherein the processor is configured to execute the machine-executable instructions to implement the method for determining object features as described above.

[0013] As can be seen from the above technical solutions, in this embodiment, a target network model can be trained based on the target object features corresponding to keyframes and the text features corresponding to sample text. This target network model can then be used to detect objects of interest to the user in the video, achieving video object localization under weak supervision. This means that even with only video and text (natural language description) and no fine-grained object annotation information, a target network model can still be trained. During the training of the target network model, fine-grained spatiotemporal annotation of objects in the sample video is not required. Instead, the training process is directly completed based on the target object features and text features, thus avoiding significant manual costs. The training of the target network model can be completed in a short time (i.e., without the large time consumption of manual annotation) and with minimal resource consumption (i.e., without the large resource consumption of manual annotation), resulting in short training time, fast training speed, and low resource consumption. By performing comprehensive comparative learning of objects in the video in both time and space, spatiotemporal ambiguity problems are reduced, significantly improving the ability to locate objects in the video based on natural language and enhancing the discrimination capability. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a method for determining object features in one embodiment of this application.

[0015] Figure 2 This is a flowchart illustrating a method for determining object features in one embodiment of this application.

[0016] Figure 3 This is a schematic diagram of the structure of an object feature determination device in one embodiment of this application;

[0017] Figure 4 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0018] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0019] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0020] This application proposes a method for determining object features, see [link to relevant documentation]. Figure 1 The diagram shown illustrates a method for determining object features, which may include the following steps:

[0021] Step 101: Obtain multiple target object features corresponding to keyframes (such as each keyframe in multiple keyframes) in the sample video, and obtain the text features corresponding to the sample text in the sample video.

[0022] For example, obtaining multiple target object features corresponding to keyframes in a sample video may include, but is not limited to: for each keyframe in the sample video, determining multiple initial object features corresponding to that keyframe; for each initial object feature, determining an attention score corresponding to that initial object feature based on the initial object feature, configured normalization parameters, and each initial object feature corresponding to that keyframe; and for each keyframe in the sample video, determining a target object feature corresponding to that initial object feature based on the initial object feature and the attention score corresponding to each initial object feature corresponding to that keyframe.

[0023] Step 102: Based on the text feature and the multiple target object features corresponding to the keyframe, determine the spatial localization score and temporal localization score between the text feature and the keyframe.

[0024] For example, determining the spatial localization score between the text feature and the keyframe based on the text feature and multiple target object features corresponding to the keyframe may include, but is not limited to: determining the semantic similarity between the text feature and each target object feature corresponding to the keyframe; selecting the maximum semantic similarity from all determined semantic similarities; and determining the spatial localization score between the text feature and the keyframe based on the maximum semantic similarity, such as using the maximum semantic similarity as the spatial localization score.

[0025] For example, based on the text feature and multiple target object features corresponding to the keyframe, the temporal localization score between the text feature and the keyframe can be determined, including but not limited to: determining the frame feature corresponding to the keyframe based on multiple target object features, and concatenating and fusing the sentence vector corresponding to the sample text (i.e., extracting the sentence vector from the sample text), the text feature, and the frame feature corresponding to the keyframe to obtain a fused feature. Based on the fused feature, the confidence score of the occurrence of the target object is calculated, and the confidence score is normalized in the time dimension to obtain the temporal localization score between the text feature and the keyframe.

[0026] Step 103: Determine the first loss value corresponding to the sample video based on the spatial positioning score and the temporal positioning score.

[0027] For example, determining the first loss value corresponding to the sample video based on the spatial localization score and the temporal localization score may include, but is not limited to: determining a first global matching score based on the spatial localization score and the temporal localization score corresponding to the positive sample video and the positive sample text, that is, using the positive sample video and the positive sample text as the sample video and sample text in step 101 to obtain the spatial localization score and the temporal localization score, and determining the first global matching score based on the spatial localization score and the temporal localization score. Also, determining a second global matching score based on the spatial localization score and the temporal localization score corresponding to the positive sample video and the negative sample text, that is, using the positive sample video and the negative sample text as the sample video and sample text in step 101 to obtain the spatial localization score and the temporal localization score, and determining the second global matching score based on the spatial localization score and the temporal localization score. Finally, determining a third global matching score based on the spatial localization score and the temporal localization score corresponding to the negative sample video and the positive sample text, that is, using the negative sample video and the positive sample text as the sample video and sample text in step 101 to obtain the spatial localization score and the temporal localization score, and determining the third global matching score based on the spatial localization score and the temporal localization score. Based on this, the first loss value corresponding to the sample video can be determined based on the first global matching score, the second global matching score, and the third global matching score.

[0028] Step 104: For each keyframe in the sample video, select representative object features from the multiple target object features corresponding to that keyframe, and determine the feature effect score corresponding to the representative object features.

[0029] For example, determining the feature effect score corresponding to a representative object feature may include, but is not limited to: selecting a representative adversarial object feature corresponding to the representative object feature from the object feature memory pool, and selecting a non-representative adversarial object feature corresponding to the non-representative object feature of the keyframe from the object feature memory pool. Based on this, the feature effect score can be determined based on the representative object feature, the representative adversarial object feature, the non-representative object feature, and the non-representative adversarial object feature.

[0030] For example, the object feature memory pool may include multiple object feature memory vectors. Representative adversarial object features corresponding to representative object features are selected from the object feature memory pool, and non-representative adversarial object features corresponding to non-representative object features for the keyframe are selected from the object feature memory pool. This may include, but is not limited to: determining the similarity between the representative object feature and each object feature memory vector, and determining the object feature memory vector with the highest similarity as the representative adversarial object feature; and determining the similarity between the non-representative object feature and each object feature memory vector, and determining the object feature memory vector with the highest similarity as the non-representative adversarial object feature.

[0031] Step 105: For multiple keyframes in the sample video, select representative keyframes from the multiple keyframes in the sample video and determine the frame effect score corresponding to the representative keyframes.

[0032] For example, determining the frame effect score corresponding to a representative keyframe may include, but is not limited to: selecting representative adversarial frame features corresponding to the frame features of the representative keyframe from the frame feature memory pool, and selecting non-representative adversarial frame features corresponding to the frame features of non-representative keyframes from the frame feature memory pool. Based on this, the frame effect score can be determined based on the frame features of the representative keyframe, the representative adversarial frame features, the frame features of the non-representative keyframe, and the non-representative adversarial frame features.

[0033] For example, the frame feature memory pool may include multiple frame feature memory vectors. Selecting representative adversarial frame features corresponding to the frame features of the representative keyframe from the frame feature memory pool, and selecting non-representative adversarial frame features corresponding to the frame features of non-representative keyframes from the frame feature memory pool, may include, but is not limited to: determining the similarity between the frame features of the representative keyframe and each frame feature memory vector, and determining the frame feature memory vector corresponding to the maximum similarity as the representative adversarial frame feature; and determining the similarity between the frame features of the non-representative keyframe and each frame feature memory vector, and determining the frame feature memory vector corresponding to the maximum similarity as the non-representative adversarial frame feature.

[0034] Step 106: Determine the second loss value corresponding to the sample video based on the feature effect score and frame effect score.

[0035] Step 107: Determine the target loss value based on the first loss value and the second loss value, and train the target network model based on the target loss value. The target network model is used to determine the object features corresponding to the video to be detected.

[0036] In one possible implementation, for each object feature memory vector in the object feature memory pool, a candidate object feature corresponding to that object feature memory vector is determined. The candidate object feature can be a representative object feature selected as a representative adversarial object feature, or a non-representative object feature selected as a non-representative adversarial object feature. Then, an updated object feature memory vector is determined based on the object feature memory vector and the candidate object feature, and the updated object feature memory vector replaces the object feature memory vector in the object feature memory pool.

[0037] In one possible implementation, for each frame feature memory vector in the frame feature memory pool, a candidate frame feature corresponding to that frame feature memory vector is determined. This candidate frame feature can be a frame feature that selects the frame feature memory vector as a representative adversarial frame feature, or it can be a frame feature that selects the frame feature memory vector as a non-representative adversarial frame feature. Then, an updated frame feature memory vector can be determined based on the frame feature memory vector and the candidate frame feature, and the updated frame feature memory vector can replace the original frame feature memory vector in the frame feature memory pool.

[0038] As can be seen from the above technical solutions, in this embodiment, a target network model can be trained based on the target object features corresponding to keyframes and the text features corresponding to sample text. This target network model can then be used to detect objects of interest to the user in the video, achieving video object localization under weak supervision. This means that even with only video and text (natural language description) and no fine-grained object annotation information, a target network model can still be trained. During the training of the target network model, fine-grained spatiotemporal annotation of objects in the sample video is not required. Instead, the training process is directly completed based on the target object features and text features, thus avoiding significant manual costs. The training of the target network model can be completed in a short time (i.e., without the large time consumption of manual annotation) and with minimal resource consumption (i.e., without the large resource consumption of manual annotation), resulting in short training time, fast training speed, and low resource consumption. By performing comprehensive comparative learning of objects in the video in both time and space, spatiotemporal ambiguity problems are reduced, significantly improving the ability to locate objects in the video based on natural language and enhancing the discrimination capability.

[0039] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0040] To accurately retrieve objects of interest from videos, a network model can be trained to detect these objects. However, training this model requires fine-grained spatiotemporal annotation of the objects in the sample videos. This includes labeling the frames containing the objects and their locations within those frames. This process demands significant manual effort, requires substantial server resources, and takes considerable time, resulting in slow annotation speeds and high resource consumption.

[0041] In response to the above findings, this application proposes a method for determining object features. Based on contrastive learning, this method can determine object features in videos under weak supervision. That is, even with only video and corresponding text (such as natural language description) and no fine-grained object annotation information, a target network model can still be trained, and the object features can be determined through the target network module.

[0042] In this embodiment, an efficient weakly supervised video object localization method based on comparative learning in time and space is proposed. This method can efficiently and accurately locate objects in videos based on natural language, avoiding temporal and spatial ambiguity issues and solving the inherent spatiotemporal ambiguity problem. It can significantly improve the ability to locate objects in videos based on natural language. By performing comprehensive comparative learning on objects in videos in time and space, the spatiotemporal ambiguity problem is reduced, and the model's discrimination ability is improved.

[0043] This application proposes a method for determining object features, see [link to relevant documentation]. Figure 2 The diagram shown illustrates a method for determining object features, which may include the following steps:

[0044] Step 201: Obtain the sample video and the corresponding sample text.

[0045] For example, the sample video can be a tutorial video or other types of video. There are no restrictions on the type of sample video, such as cooking videos, craft videos, etc.

[0046] The objects in the sample video can be of any type. For example, in cooking videos, the objects can be ingredients, cooking tools, etc.

[0047] For example, the sample video can be any video file, while the sample text is a natural language description of the sample video. For example, if the sample video is a video of "scrambled eggs with tomatoes", the sample text is used to describe how to make "scrambled eggs with tomatoes", such as the actions and steps involved.

[0048] Step 202: Select multiple keyframes from the sample video. For each keyframe in the sample video, determine multiple initial object features corresponding to that keyframe, such as K initial object features for each keyframe.

[0049] For example, a sample video can include a large number of video frames. All video frames in the sample video can be sampled at a certain sampling rate to obtain keyframes. For example, assuming a sampling rate of 20%, frames 1, 6, 11, 16, etc. in the sample video can be used as keyframes. Similarly, assuming a sampling rate of 10%, frames 1, 11, 21, 31, etc. in the sample video can be used as keyframes, and so on.

[0050] For each keyframe in the sample video, the keyframe can be input into a trained video object feature extractor. The video object feature extractor extracts multiple object features from the keyframe and outputs multiple object features. For ease of differentiation, the object features output by the video object feature extractor are recorded as initial object features. In this way, for each keyframe in the sample video, by inputting the keyframe into the video object feature extractor, multiple initial object features corresponding to the keyframe can be obtained.

[0051] For example, the initial object features can be denoted as This is used to represent the nth initial object feature corresponding to the tth keyframe, where n ranges from 1 to N, indicating that there are a total of N initial object features corresponding to the keyframe, and t ranges from 1 to T, indicating that there are a total of T keyframes corresponding to the sample video.

[0052] For example, a video object feature extractor is a network model used to extract multiple initial object features from keyframes. It can be a neural network model or a deep learning model, etc., without limitation. For instance, a video object feature extractor could be a Faster R-CNN (Faster Region Convolutional Neural Network) based on VGG-16 (Visual Geometry Group-16) to extract features and locations of objects of interest. The video object feature extractor can be trained on Visual Genome (a large-scale image semantic understanding dataset) or other training datasets; the training process is not limited.

[0053] Step 203: Obtain the text features corresponding to the sample text of the sample video.

[0054] For example, the sample text can be input into a trained text feature extractor, which will extract multiple text features from the sample text and output these multiple text features. Thus, by inputting the sample text into the text feature extractor, multiple text features can be obtained. For instance, the text features can be denoted as e. i e i Used to represent the i-th text feature output by the text feature extractor.

[0055] For example, a text feature extractor is a network model used to extract multiple text features from sample text. It can be a neural network model or a deep learning model, etc., without limitation. For example, a text feature extractor can be a GloVe model, which is used to extract text features of objects described in natural language.

[0056] Step 204: For each keyframe in the sample video, determine the target object features corresponding to each initial object feature of that keyframe, such as K initial object features corresponding to K target object features.

[0057] For example, each initial object feature corresponding to the keyframe can be input into the initial network model to be trained (in this embodiment, the initial network model needs to be trained to obtain the trained target network model), and the initial network model can determine the target object feature corresponding to each initial object feature.

[0058] For example, for each initial object feature corresponding to this keyframe, using the initial object features For example, the initial object features can be determined using the following steps. Corresponding target object features

[0059] Step 2041: Based on initial object features The configured normalization parameters and each initial object feature corresponding to this keyframe are used to determine the initial object features. The corresponding attention score.

[0060] For example, the initial object features can be determined using the following formula (1). The corresponding attention score, of course, formula (1) is just an example, and there is no restriction on how the attention score is determined.

[0061]

[0062] In formula (1), α n Represents the initial object features The corresponding attention score, where exp represents the natural logarithm operation. σ represents the initial object features, and σ represents the configured normalization parameter, which can be configured empirically. This represents all the initial object features corresponding to the keyframe (i.e., N initial object features), where n′ ranges from 1 to N, indicating that the keyframe corresponds to a total of N initial object features. This represents the first initial object feature in the t-th keyframe. This represents the second initial object feature in the t-th keyframe, and so on. W represents the Nth initial object feature in the t-th keyframe. q and W k W represents the learnable weight matrix, which is the network weight of the initial network model. q and W kThe initial values ​​can be configured empirically, that is, the initial network weights of the initial network model. During the training process of the initial network model, this is actually the process of adjusting W. q and W k The equal weight matrix is ​​adjusted and optimized to improve the performance of the initial network model.

[0063] In summary, it can be seen that based on the initial object features The configured normalization parameter σ, and the initial object features corresponding to this keyframe. Initial object features can be determined The corresponding attention score α n .

[0064] Step 2042: Based on initial object features The attention score corresponding to each initial object feature of the keyframe is used to determine the initial object features. The corresponding target object features.

[0065] For example, the initial object features can be determined using the following formula (2). The corresponding target object features. Of course, formula (2) is just an example, and there are no restrictions on the way to determine the target object features.

[0066]

[0067] In formula (2), Represents the initial object features Corresponding target object features α represents the initial object characteristics, tanh represents the hyperbolic tangent function operation, and α n′ Let α represent the attention score corresponding to all initial object features (i.e., N initial object features) for the keyframe. The value of n′ ranges from 1 to N, indicating that the keyframe corresponds to a total of N initial object features. 1′ α represents the attention score corresponding to the first initial object feature. 2′ This represents the attention score corresponding to the second initial object feature, and so on, α N′ W represents the attention score corresponding to the Nth initial object feature. v W represents the learnable weight matrix, which is the network weight of the initial network model. v The initial values ​​can be configured empirically, that is, the initial network weights of the initial network model. During the training process of the initial network model, this is actually the process of adjusting W. v The equal weight matrix is ​​adjusted and optimized.

[0068] In summary, it can be seen that based on the initial object features Each initial object feature corresponding to a keyframe The corresponding attention score α n′ It can determine the initial object features Corresponding target object features

[0069] In summary, in this embodiment, for all initial object features corresponding to each keyframe, an attention mechanism can be used for feature fusion, enabling all objects within that keyframe to perceive each other and obtain the target object features corresponding to each initial object feature. Specifically, when using the attention mechanism for feature fusion, each initial object feature in the t-th keyframe can be... As a query, and other initial object features As key and value, based on the attention score α between query and key n′ The value is weighted and then compared with the initial object features after passing through the tanh activation function. By performing residual connections, we can obtain the target object features after attention fusion.

[0070] Step 205: For each keyframe in the sample video, based on the text features and the multiple target object features corresponding to the keyframe, determine the spatial localization score between the text features and the keyframe.

[0071] For example, after the initial network model obtains multiple target object features corresponding to the keyframes, the text features and multiple target object features corresponding to the keyframes can be input into the initial network model to be trained, and the initial network model can determine the spatial localization score between the text features and the keyframes.

[0072] For example, based on text features e i Multiple target object features corresponding to the t-th keyframe Text features e can be determined i The spatial localization score between the first text feature and each keyframe (there are T keyframes in total) can be determined. For example, the spatial localization score between the first text feature and each keyframe can be determined, the spatial localization score between the second text feature and each keyframe can be determined, and so on.

[0073] In one possible implementation, text feature e i Taking the t-th keyframe as an example, the text feature e can be determined using the following steps. i Spatial localization score between the t-th keyframe and the t-th keyframe:

[0074] Step 2051: Determine the semantic similarity between the text features and the features of each target object corresponding to the keyframe.

[0075] For example, the semantic similarity between text features and target object features can be determined using the following formula (3). Of course, formula (3) is just an example, and there are no restrictions on the way to determine this semantic similarity.

[0076]

[0077] In formula (3), e i Used to represent the i-th text feature Used to represent the feature of the nth target object in the t-th keyframe, || is used to represent the modulo operation of the orientation quantity. Used to represent the semantic similarity between the i-th text feature and the n-th target object feature in the t-th keyframe.

[0078] As can be seen from formula (3), given text features e i Based on this, text features e can be calculated. i Features of each target object corresponding to the t-th keyframe semantic similarity between

[0079] Step 2052: Select the maximum semantic similarity from all the determined semantic similarities.

[0080] For example, based on text feature e i Features of each target object corresponding to the t-th keyframe semantic similarity between The maximum semantic similarity can be selected from these semantic similarities. That is, the object is located in the t-th keyframe using the maximum semantic similarity, and this object is used to represent the t-th keyframe.

[0081] Step 2053: Determine the spatial localization score between text features and keyframes based on the maximum semantic similarity. For example, the maximum semantic similarity can be used as the spatial localization score between text features and keyframes. In other words, the maximum semantic similarity can represent the spatial localization score corresponding to the keyframe.

[0082] For example, the spatial localization score between text features and keyframes can be determined using the following formula (4). Of course, formula (4) is just an example, and there are no restrictions on how the spatial localization score is determined.

[0083]

[0084] In formula (4), This is used to represent the semantic similarity between the i-th text feature and the n-th target object feature in the t-th keyframe. This represents the maximum semantic similarity among all semantic similarities. This represents the spatial localization score between the i-th text feature and the t-th keyframe.

[0085] In summary, spatial localization of objects can be performed within each keyframe to obtain the spatial localization score between each text feature and the keyframe. For example, the spatial localization score between the first text feature and the keyframe can be determined, the spatial localization score between the second text feature and the keyframe can be determined, and so on.

[0086] Step 206: For each keyframe in the sample video, based on the text features and the multiple target object features corresponding to the keyframe, determine the temporal localization score between the text features and the keyframe.

[0087] For example, after the initial network model obtains multiple target object features corresponding to the keyframes, the text features and multiple target object features corresponding to the keyframes can be input into the initial network model to be trained, and the initial network model can determine the temporal localization score between the text features and the keyframes.

[0088] For example, based on text features e i Multiple target object features corresponding to the t-th keyframe Text features e can be determined i The temporal positioning score between the first text feature and each keyframe (there are a total of T keyframes) can be determined. For example, the temporal positioning score between the first text feature and each keyframe can be determined, the temporal positioning score between the second text feature and each keyframe can be determined, and so on.

[0089] In one possible implementation, text feature e i Taking the t-th keyframe as an example, the text feature e can be determined using the following steps. i Temporal positioning score between the t-th keyframe and the t-th keyframe:

[0090] Step 2061: Generate fused features based on the sentence vector, text features and frame features corresponding to the sample text and the keyframes, such as by concatenating and fusing the sentence vector, text features and frame features to obtain fused features.

[0091] For example, based on sentence vectors, text features and frame features, the following formula (5) can be used to determine the fusion features. Of course, formula (5) is just an example, and there are no restrictions on the way to determine the fusion features.

[0092]

[0093] In formula (5), s represents the sentence vector corresponding to the sample text. For example, based on the sample text corresponding to the sample video, the sentence vector in the sample text can be extracted, and there are no restrictions on this process. i f represents the i-th text feature. t This represents the frame feature corresponding to the t-th keyframe. The frame feature f corresponding to the t-th keyframe can be determined based on multiple target object features corresponding to the t-th keyframe. t For example, we can perform max pooling on the multiple target object features corresponding to the t-th keyframe to obtain the frame feature f corresponding to the t-th keyframe. t . This represents the fused feature between the i-th text feature and the t-th keyframe.

[0094] Step 2062: Determine the temporal localization score between text features and keyframes based on the fused features. For example, calculate the confidence score of the occurrence of the target object based on the fused features, and normalize the confidence score in the time dimension to obtain the temporal localization score between text features and keyframes.

[0095] For example, the temporal localization score between text features and keyframes can be determined using the following formula (6). Of course, formula (6) is just an example, and there are no restrictions on how the temporal localization score is determined.

[0096]

[0097] In formula (6), This represents the fused feature between the i-th text feature and the t-th keyframe. W represents the temporal localization score between the i-th text feature and the t-th keyframe. a and W h W represents the learnable weight matrix, which is the network weight of the initial network model. a and W h The initial values ​​are configured based on experience; that is, the initial network weights of the initial network model. During the training process of the initial network model, this is actually the process of adjusting W. a and W h The equal-weighted matrix is ​​adjusted and optimized. h This represents the configured bias vector, which can be configured empirically. `tanh` represents the hyperbolic tangent function operation, and `softmax` represents the softmax function. The softmax function is a logistic regression function used to map the output to the (0, 1) interval, which can be understood as a probability (i.e., time-positioning score). It is a probability), which is normalized according to the time dimension in formula (6).

[0098] In formula (6), This represents the confidence level that the target object has appeared, i.e., based on fused features. Calculate the confidence score for the presence of the target object. Softmax represents the normalization of this confidence score along the time dimension, yielding a temporal localization score between text features and keyframes.

[0099] For example, since the target object in the video may not appear in every keyframe, the probability of the object appearing in each keyframe can be calculated along the time dimension. For instance, first, the sentence vector s and text features e... i and frame features f t Serial fusion yields fusion characteristics. Then, a single-layer perceptron is used to calculate the confidence score of the appearance of the target object in the keyframe, and the confidence score is normalized in the time dimension to obtain the temporal localization score. For example, the time positioning score can be calculated using formula (6).

[0100] In summary, time localization can be performed on keyframes at all frame levels within the video to obtain the time localization score between each text feature and the keyframe. For example, the time localization score between the first text feature and the keyframe can be determined, the time localization score between the second text feature and the keyframe can be determined, and so on.

[0101] Step 207: Determine the first loss value corresponding to the sample video based on the spatial localization score and the temporal localization score. For example, the first loss value can be determined based on the spatial localization score between each text feature and each keyframe, as well as the temporal localization score between each text feature and each keyframe.

[0102] In one possible implementation, the first loss value can be determined using the following steps:

[0103] Step 2071: Determine the first global matching score based on the spatial and temporal localization scores corresponding to the positive sample video and positive sample text. For example, in step 201, positive sample video and positive sample text are obtained. The positive sample video is a correct video, and the positive sample text is correct text. Based on the positive sample video and positive sample text, steps 202-206 are performed to obtain the spatial and temporal localization scores, and then the first global matching score is determined based on the spatial and temporal localization scores.

[0104] For example, the first global matching score can be determined using the following formula (7). Of course, formula (7) is just an example, and there are no restrictions on how the first global matching score is determined.

[0105]

[0106] In formula (7), Let represent the spatial localization score between the i-th text feature and the t-th keyframe. e represents the temporal localization score between the i-th text feature and the t-th keyframe. The value of t ranges from 1 to T, indicating that the sample video corresponds to a total of T keyframes. i ∈S indicates that there are a total of S text features, e i It is the i-th text feature among S text features, and e i This requires each of the S text features in sequence. sv represents the first global matching score, s represents the positive sample video, and v represents the positive sample text.

[0107] Step 2072: Determine the second global matching score based on the spatial and temporal localization scores corresponding to the positive sample video and negative sample text. For example, in step 201, positive sample video and negative sample text are obtained. The positive sample video is a correct video, while the negative sample text is incorrect text. Based on the positive sample video and negative sample text, steps 202-206 are performed to obtain the spatial and temporal localization scores, and then the second global matching score is determined based on the spatial and temporal localization scores.

[0108] For example, formula (7) can be used to determine the second global matching score, which differs from the first global matching score in that... and The spatial and temporal localization scores for positive sample videos and negative sample texts are represented by f, and the second global matching score is denoted as f. sv′ s represents positive sample video, and v′ represents negative sample text.

[0109] Step 2073: Determine the third global matching score based on the spatial and temporal localization scores corresponding to the negative sample video and the positive sample text. For example, in step 201, negative sample videos and positive sample text are obtained. The negative sample videos are incorrect videos, while the positive sample texts are correct texts. Based on the negative sample videos and positive sample texts, steps 202-206 are performed to obtain the spatial and temporal localization scores, and then the third global matching score is determined based on the spatial and temporal localization scores.

[0110] For example, formula (7) can be used to determine the third global matching score, which differs from the first global matching score in that... and The spatial and temporal localization scores represent the scores for negative sample videos and positive sample text, respectively. The third global matching score is denoted as f. s′vs′ represents negative sample video, and v represents positive sample text.

[0111] Step 2074: Determine the first loss value corresponding to the sample video based on the first global matching score, the second global matching score, and the third global matching score. For example, the first loss value can be determined using the following formula (8). Of course, formula (8) is just an example, and there are no restrictions on how the first loss value is determined.

[0112]

[0113] In formula (8), f sv′ f represents the second global matching score. sv f represents the first global match score. s′v Indicates the third global matching score. This represents the first loss value. l(x) = log(1 + exp(x / η)), which represents the logistic function (S-shaped function), and η represents the hyperparameter, which can be configured empirically.

[0114] In summary, although only video and text supervision information is available during training, by combining the spatial localization information of keyframes at the frame level and the temporal localization information of all keyframes, the overall video-level semantic similarity can be obtained for model training. Finally, the ranking loss is used to optimize the video-level semantic similarity score, and the loss function of formula (8) is constructed. The first loss value is determined based on the first global matching score, the second global matching score and the third global matching score, and the model is trained based on the first loss value.

[0115] Step 208: For each keyframe in the sample video, select a representative object feature corresponding to that keyframe from multiple target object features. Also, for multiple keyframes in the sample video, select a representative keyframe from the multiple keyframes in the sample video.

[0116] For example, multiple target object features corresponding to a keyframe can be input into the initial network model, which then selects a representative object feature from these features. Similarly, multiple keyframes from a sample video can be input into the initial network model, which then selects a representative keyframe from these keyframes.

[0117] For example, in order to reduce spatiotemporal ambiguity and make the model focus on robust and accurate spatial object localization and temporal frame localization, representative objects and video frames can be selected in space and time, that is, representative object features can be selected from multiple target object features, and representative key frames can be selected from multiple key frames.

[0118] For example, spatially, based on the semantic similarity between the text features and each target object feature corresponding to the keyframe (see step 205 for the method of determining semantic similarity), the multiple target object features corresponding to the keyframe are sorted, and starting from the target object feature with the highest semantic similarity, the top H target object features are selected as the representative object features corresponding to the keyframe. The set of representative object features is denoted as M. That is, the set M includes the H representative object features corresponding to the keyframe.

[0119] For example, in terms of time, the temporal localization score is based on the text features and each keyframe in the sample video (the method for determining the temporal localization score is described in step 206; the temporal localization score represents the probability of the target object appearing in the keyframe, and is denoted as...). The sample video is sorted by multiple keyframes, and the first A keyframes with the highest time positioning score are selected as representative keyframes. The set of representative keyframes is denoted as G, which includes A representative keyframes.

[0120] Step 209: Select representative adversarial object features corresponding to representative object features from the object feature memory pool, and select non-representative adversarial object features corresponding to non-representative object features from the object feature memory pool.

[0121] For example, after obtaining representative adversarial object features and non-representative object features, the initial network model can select representative adversarial object features corresponding to representative object features from the object feature memory pool, and select non-representative adversarial object features corresponding to non-representative object features from the object feature memory pool.

[0122] For example, for multiple target object features corresponding to a keyframe, after selecting H target object features as representative object features, the remaining target object features are used as non-representative object features corresponding to that keyframe. The object feature memory pool can include multiple object feature memory vectors. Object feature memory vectors corresponding to representative object features can be selected from the object feature memory pool, and object feature memory vectors corresponding to non-representative object features can also be selected. For ease of distinction, the object feature memory vectors corresponding to representative object features can be denoted as representative adversarial object features, and the object feature memory vectors corresponding to non-representative object features can be denoted as non-representative adversarial object features.

[0123] For example, in a model with less spatiotemporal ambiguity, the overall semantic similarity score output by the model should be more relevant to representative objects and representative keyframes, and less relevant to the remaining non-representative objects and non-representative keyframes. However, due to the lack of fine-grained supervision information, the model cannot accurately identify representative objects and representative keyframes, nor can it measure the relevance.

[0124] In order to enable the model to possess the above-mentioned capabilities in such a noisy environment (the overall semantic similarity score output by the model is more relevant to representative objects and representative keyframes, and less relevant to the remaining non-representative objects and non-representative keyframes), this embodiment adopts an adversarial contrastive learning strategy so that the model can approximately possess the above-mentioned capabilities even without precisely knowing the representative objects and representative keyframes.

[0125] Based on this, in this embodiment, a spatial memory pool network can be pre-constructed. This spatial memory pool network can be called an object feature memory pool, which can include multiple object feature memory vectors, denoted as m. j Object feature memory vector m j Let represent the j-th object feature memory vector in the object feature memory pool. The object feature memory pool is used to provide adversarial examples for representative and non-representative object features. Adversarial examples for representative object features are called representative adversarial object features, and adversarial examples for non-representative object features are called non-representative adversarial object features.

[0126] For example, multiple object features can be randomly generated as object feature memory vectors (i.e., multiple object features from the training set can be randomly selected as the initialization vectors of the object feature memory vectors), and these multiple object features can be stored in the object feature memory pool. During model operation, the object feature memory vectors in the object feature memory pool can be updated; the update process for the object feature memory vectors is described in subsequent embodiments.

[0127] For example, when selecting representative adversarial object features corresponding to representative object features from the object feature memory pool, the similarity between the representative object feature and each object feature memory vector in the object feature memory pool can be determined, and the object feature memory vector corresponding to the maximum similarity can be determined as the representative adversarial object feature. For example, the representative adversarial object feature can be determined using the following formula (9). Of course, formula (9) is just an example, and there are no restrictions on the way the representative adversarial object feature is determined.

[0128]

[0129] In formula (9), m represents the characteristics of a representative object.j This represents the object feature memory vector in the object feature memory pool. Represents representative object features and object feature memory vector m j Similarity between them This represents the memory vector of object features corresponding to the maximum similarity. Representing the characteristics of representative objects As can be seen from formula (9), the representative adversarial object feature is the memory vector of the object feature corresponding to the maximum similarity.

[0130] For example, when selecting non-representative adversarial object features corresponding to non-representative object features from the object feature memory pool, the similarity between the non-representative object features and each object feature memory vector in the object feature memory pool can be determined, and the object feature memory vector corresponding to the maximum similarity can be determined as the non-representative adversarial object feature. The determination method is similar to formula (9), and will not be repeated here.

[0131] Step 210: Select representative adversarial frame features corresponding to the frame features of representative keyframes from the frame feature memory pool, and select non-representative adversarial frame features corresponding to the frame features of non-representative keyframes from the frame feature memory pool. For example, after obtaining representative and non-representative keyframes, the initial network model can select representative adversarial frame features corresponding to the frame features of representative keyframes from the frame feature memory pool, and select non-representative adversarial frame features corresponding to the frame features of non-representative keyframes from the frame feature memory pool.

[0132] For example, for multiple keyframes in a sample video, after selecting A keyframes as representative keyframes, the remaining keyframes are designated as non-representative keyframes. The frame feature memory pool can include multiple frame feature memory vectors. Frame feature memory vectors corresponding to representative keyframes can be selected from the pool, and frame feature memory vectors corresponding to non-representative keyframes can also be selected. For ease of distinction, the frame feature memory vectors corresponding to representative keyframes can be denoted as representative adversarial frame features, and the frame feature memory vectors corresponding to non-representative keyframes can be denoted as non-representative adversarial frame features.

[0133] For example, a temporal memory pool network can be pre-constructed, which can be called a frame feature memory pool. The frame feature memory pool can include multiple frame feature memory vectors. The frame feature memory pool is used to provide adversarial examples for representative and non-representative keyframes. The adversarial examples for representative keyframes are called representative adversarial frame features, and the adversarial examples for non-representative keyframes are called non-representative adversarial frame features.

[0134] For example, multiple frame features can be randomly generated as frame feature memory vectors, and these multiple frame features can be stored in a frame feature memory pool. During model operation, the frame feature memory vectors in the frame feature memory pool can be updated; the update process of the frame feature memory vectors is described in subsequent embodiments.

[0135] For example, when selecting representative adversarial frame features corresponding to representative keyframes from the frame feature memory pool, the similarity between the frame features of the representative keyframe (used to characterize the frame features of the representative keyframe, and the method of determining these frame features is not restricted) and each frame feature memory vector in the frame feature memory pool can be determined, and the frame feature memory vector corresponding to the maximum similarity is determined as the representative adversarial frame feature. The method of determining the representative adversarial frame feature is similar to that of formula (9), and will not be repeated here.

[0136] For example, when selecting non-representative adversarial frame features corresponding to non-representative keyframes from the frame feature memory pool, the similarity between the frame features of the non-representative keyframes and each frame feature memory vector in the frame feature memory pool can be determined, and the frame feature memory vector corresponding to the maximum similarity can be determined as the non-representative adversarial frame feature. The determination method is similar to formula (9), and will not be repeated here.

[0137] Step 211: Determine the feature effect score based on the representative object feature, the representative adversary object feature, the non-representative object feature, and the non-representative adversary object feature.

[0138] In one possible implementation, the characteristic effect score can be determined using the following steps:

[0139] Step S11: Determine the first output score corresponding to the representative object features.

[0140] For example, the first output score reflects the output of the initial network model. It can be the global matching score or other types of scores; there are no restrictions. Taking the global matching score as an example, referring to steps 201-204, multiple target object features can be obtained. These features include representative and non-representative object features. Based on these features, steps 205 and 206 are executed to obtain the spatial localization score and the temporal localization score. The global matching score can be determined based on the spatial and temporal localization scores, as shown in step 207. Thus, the global matching score f can be obtained. sv In this way, the global matching score f can be obtained. sv As the first output score, in subsequent embodiments, the first output score corresponding to the representative object features is denoted as Net(U). M ). Among them, U MTo represent any representative object feature, we denote the representative object feature as U. M .

[0141] Step S12: Determine the second output score corresponding to the representative adversarial object features.

[0142] For example, the second output score reflects the output of the initial network model and can be the global matching score. Referring to steps 201-204, multiple target object features can be obtained, including representative and non-representative object features. The representative object features need to be replaced with the corresponding representative adversarial object features. Based on the representative and non-representative object features, i.e., replacing the representative object features with representative adversarial object features, steps 205 and 206 are executed to obtain spatial localization scores and temporal localization scores. Based on the spatial and temporal localization scores, the global matching score can be determined. Thus, the global matching score can be used as the second output score. In subsequent embodiments, the second output score corresponding to the representative adversarial object feature is denoted as...

[0143] Step S13: Determine the third output score corresponding to the non-representative object features.

[0144] For example, the third output score reflects the output of the initial network model and can be the global matching score. Referring to steps 201-204, multiple target object features can be obtained, including representative and non-representative object features. By removing the representative object features and using the non-representative object features as a basis, steps 205 and 206 are performed to obtain the spatial localization score and the temporal localization score. Based on the spatial and temporal localization scores, the global matching score can be determined. Thus, the global matching score can be used as the third output score, which is denoted as... in, To represent any non-representative object feature, we denote the non-representative object feature as...

[0145] Step S14: Determine the fourth output score corresponding to the non-representative adversarial object features.

[0146] For example, the fourth output score reflects the output of the initial network model and can be the global matching score. Referring to steps 201-204, multiple target object features can be obtained, including representative and non-representative object features. By removing the representative object features and replacing the non-representative object features with their corresponding non-representative adversarial object features, steps 205 and 206 are executed based on these non-representative adversarial object features to obtain spatial and temporal localization scores. The global matching score can then be determined based on these scores. Thus, the global matching score can be used as the fourth output score, denoted as...

[0147] Step S15: Based on the first output score Second output score Third output score and the fourth output score Determine the characteristic effect score.

[0148] For example, the correlation between a certain type of sample and the overall output of the model can be measured, and the correlation can be expressed by the following formula: The difference between the first and second output scores represents the correlation between the representative object features. The above formula means that if the representative object feature U... M Replace with representative object feature U M Corresponding representative adversarial object features So how much impact will this have on the final overall output (i.e., output score) of model Net(·)? Similarly, the correlation can be expressed by the following formula: The difference between the third and fourth output scores represents the correlation between the features of non-representative objects.

[0149] Based on this correlation measurement method, it is hoped that the model will still have the property of being more correlated with the original representative object and less correlated with the remaining non-representative object for the generated adversarial samples (representative adversarial object features and non-representative adversarial object features). Therefore, a contrastive learning strategy is used to constrain these adversarial samples. For example, the following formula (10) can be used to constrain them:

[0150]

[0151] By transforming formula (10), the relationship between formula (11) and formula (12) can be obtained.

[0152]

[0153]

[0154] In summary, it can be As a characteristic effect score, the characteristic effect score is a constraint that the characteristic effect score must be less than 0.

[0155] Step 212: Determine the frame effect score based on the frame features of the representative keyframe, the representative adversarial frame features, the frame features of the non-representative keyframe, and the non-representative adversarial frame features.

[0156] In one possible implementation, the frame effect score can be determined using the following steps:

[0157] Step S21: Determine the fifth output score corresponding to the frame features of the representative keyframe.

[0158] For example, the fifth output score reflects the output of the initial network model. It can be a global matching score or other types of scores; there are no restrictions. Taking the global matching score as an example, referring to step 206, the frame features f corresponding to the keyframes can be obtained. t This allows us to obtain the frame features of representative and non-representative keyframes. Based on these features, step 206 is executed to obtain the temporal localization score. The global matching score can then be determined based on this score, as shown in step 207. sv In this way, the global matching score f can be obtained. sv As the fifth output score, the fifth output score is denoted as Net(U) G ). Among them, U G The frame features of any representative keyframe are denoted as U. G .

[0159] Step S22: Determine the sixth output score corresponding to the representative adversarial frame features.

[0160] For example, the sixth output score reflects the output of the initial network model and can be the global matching score. Referring to step 206, frame features of representative keyframes and non-representative keyframes can be obtained. The frame features of the representative keyframes need to be replaced with the corresponding representative adversarial frame features. Based on the representative adversarial frame features and the frame features of the non-representative keyframes, step 206 is executed to obtain the temporal localization score. The global matching score can then be determined based on the temporal localization score. Thus, the global matching score can be used as the sixth output score, and the sixth output score is denoted as...

[0161] Step S23: Determine the seventh output score corresponding to the frame features of the non-representative keyframes.

[0162] For example, the seventh output score reflects the output of the initial network model and can be the global matching score. Referring to step 206, frame features of representative and non-representative keyframes can be obtained. By removing the frame features of representative keyframes and using the frame features of non-representative keyframes as a basis, step 206 is executed to obtain the temporal localization score. Based on the temporal localization score, the global matching score can be determined. Thus, the global matching score is used as the seventh output score, and this seventh output score is denoted as... The frame features of any non-representative keyframe are denoted as:

[0163] Step S24: Determine the eighth output score corresponding to the non-representative adversarial frame features.

[0164] For example, the eighth output score reflects the output of the initial network model and can be the global matching score. Referring to step 206, frame features of representative and non-representative keyframes can be obtained. By removing the frame features of representative keyframes and replacing the frame features of non-representative keyframes with the corresponding non-representative adversarial frame features, step 206 is performed based on the non-representative adversarial frame features to obtain the temporal localization score. The global matching score can then be determined based on the temporal localization score. Thus, the global matching score can be used as the eighth output score, which is denoted as...

[0165] Step S25: Based on the fifth output score Net(U) G ), Sixth Output Score Seventh Output Score and the eighth output score Determine the frame effect score.

[0166] For example, correlation can be represented by the following formula: The difference between the fifth and sixth output scores represents the relevance of the representative keyframes. The above formula means how much impact replacing the frame features of the representative keyframes with the features of the representative adversarial frames would have on the final overall output of the Net(·) model. Similarly, the relevance can also be expressed by the following formula: The difference between the seventh and eighth output scores represents the relevance of non-representative keyframes. Based on this, a contrastive learning strategy can be used to constrain these adversarial examples, for example, through the following formula:

[0167]

[0168] By transforming the above formula, we can obtain the following relationship.

[0169]

[0170] In summary, it can be As a frame effect score, the frame effect score is a constraint that the frame effect score must be less than 0.

[0171] Step 213: Determine the second loss value corresponding to the sample video based on the feature effect score and frame effect score.

[0172] For example, based on the feature effect score and the frame effect score, the second loss value can be determined using the following formula (13). Of course, formula (13) is just an example, and there are no restrictions on how the second loss value is determined.

[0173]

[0174] express express The characteristic effect score. express express The frame effect score. This represents the second loss value. l(x) = log(1 + exp(x / η)), which represents the logistic function, and η represents the hyperparameter.

[0175] In summary, the second loss value can be determined based on the feature effect score and the frame effect score.

[0176] Step 214: Determine the target loss value based on the first loss value and the second loss value.

[0177] For example, the target loss value can be determined using the following formula (14). Of course, formula (14) is just an example, and there are no restrictions on how the target loss value is determined.

[0178]

[0179] This represents the first loss value. This represents the second loss value. This represents the target loss value.

[0180] Step 215: Train the target network model based on the target loss value.

[0181] For example, after obtaining the target loss value, the network weights of the initial network model can be adjusted based on the target loss value, such as using algorithms like gradient descent. There are no restrictions on this adjustment process. If the adjusted network model has converged, it is used as the target network model, thus obtaining the trained target network model. If the adjusted network model has not converged, it is used as the initial network model, and the process returns to step 204, repeating the above steps, i.e., repeatedly training the initial network model until the adjusted network model converges, thus obtaining the target network model.

[0182] In one possible implementation, after contrastive learning, in order to generate higher-quality adversarial examples in the next round, the object feature memory vectors in the object feature memory pool can be updated. That is, the object feature memory vectors in the object feature memory pool are adjusted according to the adversarial example generation situation in this round. The updating process of the object feature memory pool can include the following steps:

[0183] Step S31: For each object feature memory vector in the object feature memory pool, determine the candidate object feature corresponding to the object feature memory vector. The candidate object feature can be a representative object feature of the object feature memory vector as a representative adversarial object feature, or the candidate object feature can be a non-representative object feature of the object feature memory vector as a non-representative adversarial object feature.

[0184] For example, regarding the object feature memory vector m in the object feature memory pool j If the object feature memory vector m j As a representative object feature The corresponding representative adversarial object features, then, representative object features m, as the object feature memory vector j The corresponding candidate object features. If the object feature memory vector m j If a non-representative object feature corresponds to a non-representative object feature, then this non-representative object feature is used as the object feature memory vector m. j The corresponding candidate object features.

[0185] Step S32: Based on the object feature memory vector and the candidate object features corresponding to the object feature memory vector, determine the updated object feature memory vector, and replace the object feature memory vector in the object feature memory pool with the updated object feature memory vector, that is, update the object feature memory vector.

[0186] For example, see Equations (15) and (16) for an example of determining the updated object feature memory vector, without any restrictions on the updating process of the object feature memory vector.

[0187]

[0188]

[0189] Memorizing the object feature vector m j In other words, Representing representative or non-representative object characteristics m, as the object feature memory vector j Representative or non-representative features of adversaries therefore, m, as the object feature memory vector j Candidate object features. j Represents the object feature memory vector m j The average value of all candidate object features, where α represents the weighting coefficient, which can be configured empirically.

[0190] In one possible implementation, after contrastive learning, to generate higher-quality adversarial examples in the next round, the frame feature memory vectors in the frame feature memory pool can be updated. For example, for each frame feature memory vector in the frame feature memory pool, a candidate frame feature corresponding to that frame feature memory vector is determined. This candidate frame feature can be a frame feature that selects the frame feature memory vector as a representative adversarial frame feature, or a frame feature that selects the frame feature memory vector as a non-representative adversarial frame feature. Based on the frame feature memory vector and the candidate frame feature, an updated frame feature memory vector is determined, and the updated frame feature memory vector replaces the original frame feature memory vector in the frame feature memory pool.

[0191] In one possible implementation, after obtaining the target network model, the object features corresponding to the video to be detected can be determined based on the target network model. For example, the video to be detected and the corresponding text to be detected can be obtained. Multiple keyframes are selected from the video to be detected, and for each keyframe, multiple initial object features corresponding to that keyframe are determined, such as K initial object features for each keyframe. Additionally, the text features corresponding to the text to be detected in the video to be detected are obtained.

[0192] Based on this, multiple initial object features corresponding to each keyframe and text features corresponding to the text to be detected can be input into the target network model. For the target network model, the target object features corresponding to each initial object feature of the keyframe can be determined. Then, the semantic similarity between the text features and each target object feature of the keyframe can be determined, and the maximum semantic similarity is selected from all the determined semantic similarities. The initial object feature corresponding to the target object feature with the maximum semantic similarity is the object feature corresponding to that text feature in that keyframe, and the target network model outputs this object feature.

[0193] For example, the target network model outputs the object features corresponding to text feature 1 in keyframe 1, the object features corresponding to text feature 1 in keyframe 2, the object features corresponding to text feature 2 in keyframe 1, the object features corresponding to text feature 2 in keyframe 2, and so on. These object features are the object features corresponding to the video to be detected, that is, the target network model outputs the object features corresponding to the video to be detected.

[0194] As can be seen from the above technical solutions, in this embodiment, a target network model can be trained based on the target object features corresponding to keyframes and the text features corresponding to sample text. This target network model can then be used to detect objects of interest to the user in the video, achieving video object localization under weak supervision. This means that even with only video and text (natural language description) and no fine-grained object annotation information, a target network model can still be trained. During the training of the target network model, fine-grained spatiotemporal annotation of objects in the sample video is not required. Instead, the training process is directly completed based on the target object features and text features, thus avoiding significant manual costs. The training of the target network model can be completed in a short time (i.e., without the large time consumption of manual annotation) and with minimal resource consumption (i.e., without the large resource consumption of manual annotation), resulting in short training time, fast training speed, and low resource consumption. By performing comprehensive comparative learning of objects in the video in both time and space, spatiotemporal ambiguity problems are reduced, significantly improving the ability to locate objects in the video based on natural language and enhancing the discrimination capability.

[0195] In one possible implementation, this embodiment proposes a video object localization method, comprising:

[0196] Video object feature and text feature extraction process: A VGG-16-based Faster R-CNN pre-trained on Visual Genome is used as the video object feature extractor to extract features of objects of interest in the video. A pre-trained GloVe model is used to extract text features of the objects in the natural language description.

[0197] The framework for constructing a frame-level multi-instance learning system involves extracting keyframes from the video at a certain sampling rate. Each keyframe is treated as a packet, and the extracted objects are instances within that keyframe. An attention mechanism is used to fuse the instance features from all keyframes, and a maximum value operator is employed to select representative instance features. A temporal attention mechanism is also used to calculate the probability that all keyframes contain the target object, which is then used to aggregate the frame-level keyframes. Finally, video-level object features are obtained. These features are then cross-modal aligned with text features to obtain video-level semantic similarity, which can be directly constrained by weak annotation information.

[0198] Spatiotemporal contrastive learning is used to reduce spatiotemporal ambiguity: Following the frame-level multi-instance learning framework described above, high-scoring and low-scoring objects are selected based on the similarity of object instances within each keyframe. High-scoring and low-scoring frames are also selected based on the weights of all keyframes within the same video. Then, an adversarial contrastive learning strategy is adopted, using a memory pool to generate corresponding adversarial examples for high-scoring and low-scoring features to replace the original features. Finally, the adversarial example matching score is constrained by the contrastive learning strategy.

[0199] Based on the same concept as the method described above, this application proposes a device for determining object features, see [link to relevant documentation]. Figure 3 The diagram shown is a structural schematic of the device, which may include:

[0200] The acquisition module 31 is used to acquire multiple target object features corresponding to key frames in the sample video and acquire text features corresponding to sample text in the sample video.

[0201] The determining module 32 is configured to determine, based on the text features and the multiple target object features, the spatial localization score and the temporal localization score between the text features and the keyframe; and to determine the first loss value corresponding to the sample video based on the spatial localization score and the temporal localization score.

[0202] The determining module 32 is further configured to, for each keyframe in the sample video, select representative object features from multiple target object features corresponding to the keyframe, and determine the feature effect score corresponding to the representative object features; select representative keyframes from multiple keyframes in the sample video, and determine the frame effect score corresponding to the representative keyframes; determine a second loss value corresponding to the sample video based on the feature effect score and the frame effect score; and determine a target loss value based on the first loss value and the second loss value.

[0203] Processing module 33 is used to train a target network model based on the target loss value, wherein the target network model is used to determine the object features corresponding to the video to be detected.

[0204] For example, when the acquisition module 31 acquires multiple target object features corresponding to keyframes in the sample video, it is specifically used to: for each keyframe in the sample video, determine multiple initial object features corresponding to the keyframe; for each initial object feature, determine the attention score corresponding to the initial object feature based on the initial object feature, the configured normalization parameter, and each initial object feature corresponding to the keyframe; and determine the target object feature corresponding to the initial object feature based on the initial object feature and the attention score corresponding to each initial object feature corresponding to the keyframe.

[0205] For example, when determining the spatial positioning score between the text feature and the keyframe based on the text feature and the plurality of target object features, the determining module 32 is specifically used to: determine the semantic similarity between the text feature and each target object feature corresponding to the keyframe; select the maximum semantic similarity from all determined semantic similarities; and determine the spatial positioning score between the text feature and the keyframe based on the maximum semantic similarity.

[0206] For example, when determining the temporal localization score between the text features and the keyframe based on the text features and the multiple target object features, the determining module 32 is specifically used to: determine the frame features corresponding to the keyframe based on the multiple target object features, and concatenate and fuse the sentence vector corresponding to the sample text, the text features, and the frame features to obtain fused features; and calculate the confidence score of the occurrence of the target object based on the fused features, and normalize the confidence score in the time dimension to obtain the temporal localization score between the text features and the keyframe.

[0207] When determining the first loss value corresponding to the sample video based on the spatial positioning score and the temporal positioning score, the determining module 32 is specifically used to: determine a first global matching score based on the spatial positioning score and temporal positioning score corresponding to the positive sample video and the positive sample text; determine a second global matching score based on the spatial positioning score and temporal positioning score corresponding to the positive sample video and the negative sample text; determine a third global matching score based on the spatial positioning score and temporal positioning score corresponding to the negative sample video and the positive sample text; and determine the first loss value based on the first global matching score, the second global matching score, and the third global matching score.

[0208] For example, when determining the feature effect score corresponding to the representative object feature, the determining module 32 is specifically used to: select representative adversarial object features corresponding to the representative object feature from the object feature memory pool, and select non-representative adversarial object features corresponding to the non-representative object feature corresponding to the keyframe from the object feature memory pool; determine the feature effect score based on the representative object feature, the representative adversarial object feature, the non-representative object feature, and the non-representative adversarial object feature; when determining the frame effect score corresponding to the representative keyframe, the determining module 32 is specifically used to: select representative adversarial frame features corresponding to the frame features of the representative keyframe from the frame feature memory pool, and select non-representative adversarial frame features corresponding to the frame features of the non-representative keyframe from the frame feature memory pool; determine the frame effect score based on the frame features of the representative keyframe, the representative adversarial frame features, the non-representative keyframe, and the non-representative adversarial frame features.

[0209] For example, the object feature memory pool includes multiple object feature memory vectors. When the determining module 32 selects the representative adversarial object feature corresponding to the representative object feature from the object feature memory pool, and selects the non-representative adversarial object feature corresponding to the non-representative object feature of the keyframe from the object feature memory pool, it is specifically used to: determine the similarity between the representative object feature and each object feature memory vector, and determine the object feature memory vector corresponding to the maximum similarity as the representative adversarial object feature; determine the similarity between the non-representative object feature and each object feature memory vector, and determine the object feature memory vector corresponding to the maximum similarity as the non-representative adversarial object feature.

[0210] For example, the frame feature memory pool includes multiple frame feature memory vectors. When the determining module 31 selects representative adversarial frame features corresponding to the frame features of the representative keyframe from the frame feature memory pool, and selects non-representative adversarial frame features corresponding to the frame features of non-representative keyframes from the frame feature memory pool, it is specifically used to: determine the similarity between the frame features of the representative keyframe and each frame feature memory vector, and determine the frame feature memory vector corresponding to the maximum similarity as the representative adversarial frame feature; determine the similarity between the frame features of the non-representative keyframe and each frame feature memory vector, and determine the frame feature memory vector corresponding to the maximum similarity as the non-representative adversarial frame feature.

[0211] For example, the determining module 32 is further configured to: for each object feature memory vector in the object feature memory pool, determine a candidate object feature corresponding to the object feature memory vector, wherein the candidate object feature is a representative object feature selected as a representative adversarial object feature, or the candidate object feature is a non-representative object feature selected as a non-representative adversarial object feature; determine an updated object feature memory vector based on the object feature memory vector and the candidate object feature, and replace the object feature memory vector in the object feature memory pool with the updated object feature memory vector; furthermore, the determining module 32 is further configured to: for each frame feature memory vector in the frame feature memory pool, determine a candidate frame feature corresponding to the frame feature memory vector, wherein the candidate frame feature is a frame feature selected as a representative adversarial frame feature, or the frame feature memory vector is a non-representative adversarial frame feature; determine an updated frame feature memory vector based on the frame feature memory vector and the candidate frame feature, and replace the frame feature memory vector in the frame feature memory pool with the updated frame feature memory vector.

[0212] Based on the same concept as the above method, this application proposes an electronic device, see [link to previous application]. Figure 4 As shown, the electronic device includes a processor 41 and a machine-readable storage medium 42, the machine-readable storage medium 42 storing machine-executable instructions that can be executed by the processor 41; the processor 41 is used to execute the machine-executable instructions to implement the method for determining object features disclosed in the above examples of this application.

[0213] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the method for determining object features disclosed in the above examples of this application.

[0214] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0215] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer entity or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0216] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0217] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0218] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0219] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0220] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0221] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for determining the features of an object, characterized in that, The method includes: Multiple target object features corresponding to keyframes in a sample video are obtained, and text features corresponding to sample text in the sample video are obtained. Based on the text features and the multiple target object features, spatial localization scores and temporal localization scores between the text features and the keyframes are determined. A first loss value corresponding to the sample video is determined based on the spatial localization scores and temporal localization scores. The spatial localization score represents the maximum semantic similarity between the text features and each target object feature corresponding to the keyframe. The temporal localization score is obtained by normalizing the confidence of the occurrence of target objects in the keyframes in the temporal dimension. For each keyframe in the sample video, representative object features are selected from multiple target object features corresponding to that keyframe, and a feature effect score corresponding to the representative object features is determined. Representative keyframes are selected from multiple keyframes in the sample video, and a frame effect score corresponding to the representative keyframe is determined. A second loss value corresponding to the sample video is determined based on the feature effect score and the frame effect score. The feature effect score is determined based on the representative object features, representative adversarial object features, and non-representative object features and non-representative adversarial object features corresponding to that keyframe. The frame effect score is determined based on the frame features of the representative keyframe, representative adversarial frame features, frame features of non-representative keyframes, and non-representative adversarial frame features. A target loss value is determined based on a first loss value and a second loss value. A target network model is trained based on the target loss value. The target network model is used to determine the object features corresponding to the video to be detected. Specifically, representative adversarial object features corresponding to the representative object features are selected from the object feature memory pool, and non-representative adversarial object features corresponding to the non-representative object features of the keyframe are selected from the object feature memory pool. The object feature memory pool includes multiple object feature memory vectors. Based on the similarity between the representative object feature and each object feature memory vector, the object feature memory vector with the highest similarity is determined as the representative adversarial object feature. Similarly, based on the similarity between the non-representative object feature and each object feature memory vector, the object feature memory vector with the highest similarity is determined as the non-representative adversarial object feature. Specifically, representative adversarial frame features corresponding to the frame features of the representative keyframes are selected from the frame feature memory pool, and non-representative adversarial frame features corresponding to the frame features of non-representative keyframes are selected from the frame feature memory pool. The frame feature memory pool includes multiple frame feature memory vectors. Based on the similarity between the frame features of the representative keyframes and each frame feature memory vector, the frame feature memory vector with the highest similarity is determined as the representative adversarial frame feature. Similarly, based on the similarity between the frame features of the non-representative keyframes and each frame feature memory vector, the frame feature memory vector with the highest similarity is determined as the non-representative adversarial frame feature.

2. The method according to claim 1, characterized in that, The acquisition of multiple target object features corresponding to keyframes in the sample video includes: For each keyframe in the sample video, determine multiple initial object features corresponding to that keyframe; For each initial object feature, based on the initial object feature, the configured normalization parameters, and each initial object feature corresponding to the keyframe, the attention score corresponding to the initial object feature is determined. Based on the initial object features and the attention scores corresponding to each initial object feature in the keyframe, the target object features corresponding to the initial object features are determined.

3. The method according to claim 1, characterized in that, Based on the text features and the multiple target object features, the spatial localization score between the text features and the keyframe is determined, including: Determine the semantic similarity between the text features and the target object features corresponding to each keyframe; Select the maximum semantic similarity from all determined semantic similarities; The spatial localization score between the text features and the keyframe is determined based on the maximum semantic similarity.

4. The method according to claim 1, characterized in that, Based on the text features and the multiple target object features, the temporal localization score between the text features and the keyframe is determined, including: Based on the features of the multiple target objects, the frame features corresponding to the key frame are determined, and the sentence vector corresponding to the sample text, the text features, and the frame features are concatenated and fused to obtain the fused features; The confidence score of the occurrence of the target object is calculated based on the fused features, and the confidence score is normalized in the time dimension to obtain the temporal localization score between the text features and the keyframe.

5. The method according to claim 1, characterized in that, Determining the first loss value corresponding to the sample video based on the spatial positioning score and the temporal positioning score includes: A first global matching score is determined based on the spatial and temporal localization scores corresponding to the positive sample videos and positive sample texts. A second global matching score is determined based on the spatial and temporal localization scores corresponding to the positive sample videos and negative sample texts. A third global matching score is determined based on the spatial and temporal localization scores corresponding to the negative sample videos and positive sample texts. A first loss value is determined based on the first, second, and third global matching scores.

6. The method according to claim 1, characterized in that, The method further includes: For each object feature memory vector in the object feature memory pool, a candidate object feature corresponding to the object feature memory vector is determined. The candidate object feature is either a representative object feature selected as a representative adversarial object feature, or a non-representative object feature selected as a non-representative adversarial object feature. Based on the object feature memory vector and the candidate object feature, an updated object feature memory vector is determined, and the updated object feature memory vector replaces the object feature memory vector in the object feature memory pool. For each frame feature memory vector in the frame feature memory pool, a candidate frame feature corresponding to the frame feature memory vector is determined. The candidate frame feature is either a frame feature that selects the frame feature memory vector as a representative adversarial frame feature, or a frame feature that selects the frame feature memory vector as a non-representative adversarial frame feature. Based on the frame feature memory vector and the candidate frame feature, an updated frame feature memory vector is determined, and the updated frame feature memory vector replaces the frame feature memory vector in the frame feature memory pool.

7. A device for determining the features of an object, characterized in that, The device includes: The acquisition module is used to acquire multiple target object features corresponding to key frames in the sample video, and to acquire text features corresponding to sample text in the sample video. A determining module is configured to determine, based on the text features and the multiple target object features, a spatial localization score and a temporal localization score between the text features and the keyframes; and to determine a first loss value corresponding to the sample video based on the spatial localization score and the temporal localization score; wherein the spatial localization score represents the maximum semantic similarity between the text features and each target object feature corresponding to the keyframes; and wherein the confidence of the occurrence of target objects in the keyframes is normalized in the temporal dimension to obtain the temporal localization score. The determining module is further configured to, for each keyframe in the sample video, select representative object features from multiple target object features corresponding to that keyframe, and determine the feature effect score corresponding to the representative object features; select representative keyframes from multiple keyframes in the sample video, and determine the frame effect score corresponding to the representative keyframes; determine a second loss value corresponding to the sample video based on the feature effect score and the frame effect score; and determine a target loss value based on the first loss value and the second loss value; wherein, the feature effect score is determined based on the representative object features, representative adversarial object features, and non-representative object features and non-representative adversarial object features corresponding to that keyframe; and the frame effect score is determined based on the frame features of the representative keyframe, representative adversarial frame features, frame features of the non-representative keyframe, and non-representative adversarial frame features. The processing module is used to train a target network model based on the target loss value, wherein the target network model is used to determine the object features corresponding to the video to be detected; Specifically, representative adversarial object features corresponding to the representative object features are selected from the object feature memory pool, and non-representative adversarial object features corresponding to the non-representative object features of the keyframe are selected from the object feature memory pool. The object feature memory pool includes multiple object feature memory vectors. Based on the similarity between the representative object feature and each object feature memory vector, the object feature memory vector with the highest similarity is determined as the representative adversarial object feature. Similarly, based on the similarity between the non-representative object feature and each object feature memory vector, the object feature memory vector with the highest similarity is determined as the non-representative adversarial object feature. Specifically, representative adversarial frame features corresponding to the frame features of the representative keyframes are selected from the frame feature memory pool, and non-representative adversarial frame features corresponding to the frame features of non-representative keyframes are selected from the frame feature memory pool. The frame feature memory pool includes multiple frame feature memory vectors. Based on the similarity between the frame features of the representative keyframes and each frame feature memory vector, the frame feature memory vector with the highest similarity is determined as the representative adversarial frame feature. Similarly, based on the similarity between the frame features of the non-representative keyframes and each frame feature memory vector, the frame feature memory vector with the highest similarity is determined as the non-representative adversarial frame feature.

8. An electronic device, characterized in that, include: A processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor; wherein the processor is configured to execute the machine-executable instructions to implement the method of any one of claims 1-6.

Images